8000 [dynamic shapes] translation validation failure under `fake_tensor_propagate_real_tensors` · Issue #156251 · pytorch/pytorch · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
[dynamic shapes] translation validation failure under fake_tensor_propagate_real_tensors #156251
Open
@StrongerXi

Description

@StrongerXi

🐛 Describe the bug

Discovered in #155863 (comment).

Repro:

import torch

@torch.compile(backend="eager", fullgraph=True)
def f(x):
    if x.eq(0.1).any().item():
        return x
    return x + 1

torch._dynamo.config.capture_scalar_outputs = True
torch.fx.experimental._config.translation_validation = True
torch._functorch.config.fake_tensor_propagate_real_tensors = True
f(torch.randn(2))

Error logs

W0617 16:19:14.454000 3219607 torch/fx/experimental/symbolic_shapes.py:7259] [0/0] propagate_real_tensors evaluate_expr(Eq(u0, 1)) -> 0
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0] Error while creating guard:
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0] Name: ''
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]     Source: shape_env
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]     Create Function: SHAPE_ENV
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]     Guard Types: None
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]     Code List: None
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]     Object Weakref: None
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]     Guarded Class Weakref: None
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0] Traceback (most recent call last):
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   File "/home/ryanguo99/pt/pytorch/torch/_guards.py", line 360, in create
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]     return self.create_fn(builder, self)
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   File "/home/ryanguo99/pt/pytorch/torch/_dynamo/guards.py", line 2092, in SHAPE_ENV
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]     python_code_parts, verbose_code_parts = _get_code_parts(
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]                                             ^^^^^^^^^^^^^^^^
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   File "/home/ryanguo99/pt/pytorch/torch/_dynamo/guards.py", line 2066, in _get_code_parts
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]     return output_graph.shape_env.produce_guards_verbose(
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   File "/home/ryanguo99/pt/pytorch/torch/fx/experimental/symbolic_shapes.py", line 5967, in produce_guards_verbose
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]     self._check_translation_validate()
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   File "/home/ryanguo99/pt/pytorch/torch/fx/experimental/symbolic_shapes.py", line 4211, in _check_translation_validate
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]     self.validator.validate()
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   File "/home/ryanguo99/pt/pytorch/torch/fx/experimental/validator.py", line 569, in validate
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]     return self._validate()
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]            ^^^^^^^^^^^^^^^^
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   File "/home/ryanguo99/pt/pytorch/torch/fx/experimental/validator.py", line 606, in _validate
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]     raise ValidationException(
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0] torch.fx.experimental.validator.ValidationException: translation validation failed.
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0] Model:
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   ==> L['x'].size()[0]: 2
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   ==> L['x'].storage_offset(): 0
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   ==> L['x'].stride()[0]: 1
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   ==> u0: True
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0] Assertions:
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   ==> (== 0 L['x'].storage_offset())
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   ==> (== 1 L['x'].stride()[0])
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   ==> (== 2 L['x'].size()[0])
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0] Target Expressions:
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   ==> (<= 0 (If u0 1 0))
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   ==> (== 0 L['x'].storage_offset())
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   ==> (== 1 L['x'].stride()[0])
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   ==> (== 2 L['x'].size()[0])
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   ==> (>= 1 (If u0 1 0))
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0] Failed Source Expressions:
E0617 16:19:14.485000 3219607 torch/_guards.py:362] [0/0]   ==> (Not u0)
E0617 16:19:14.490000 3219607 torch/_guards.py:364] [0/0] Created at:
E0617 16:19:14.490000 3219607 torch/_guards.py:364] [0/0]   File "/home/ryanguo99/pt/pytorch/torch/_dynamo/convert_frame.py", line 722, in transform
E0617 16:19:14.490000 3219607 torch/_guards.py:364] [0/0]     tracer = InstructionTranslator(
E0617 16:19:14.490000 3219607 torch/_guards.py:364] [0/0]   File "/home/ryanguo99/pt/pytorch/torch/_dynamo/symbolic_convert.py", line 3314, in __init__
E0617 16:19:14.490000 3219607 torch/_guards.py:364] [0/0]     output=OutputGraph(
E0617 16:19:14.490000 3219607 torch/_guards.py:364] [0/0]   File "/home/ryanguo99/pt/pytorch/torch/_dynamo/output_graph.py", line 444, in __init__
E0617 16:19:14.490000 3219607 torch/_guards.py:364] [0/0]     self.init_ambient_guards()
E0617 16:19:14.490000 3219607 torch/_guards.py:364] [0/0]   File "/home/ryanguo99/pt/pytorch/torch/_dynamo/output_graph.py", line 611, in init_ambient_guards
E0617 16:19:14.490000 3219607 torch/_guards.py:364] [0/0]     self.guards.add(ShapeEnvSource().make_guard(GuardBuilder.SHAPE_ENV))
Traceback (most recent call last):
  File "/home/ryanguo99/pt/scratch/cell.py", line 12, in <module>
    f(torch.randn(2))
  File "/home/ryanguo99/pt/pytorch/torch/_dynamo/eval_frame.py", line 701, in compile_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/ryanguo99/pt/pytorch/torch/_dynamo/convert_frame.py", line 1470, in __call__
    return self._torchdynamo_orig_callable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ryanguo99/pt/pytorch/torch/_dynamo/convert_frame.py", line 626, in __call__
    return _compile(
           ^^^^^^^^^
  File "/home/ryanguo99/pt/pytorch/torch/_dynamo/convert_frame.py", line 1093, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ryanguo99/pt/pytorch/torch/_utils_internal.py", line 97, in wrapper_function
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ryanguo99/pt/pytorch/torch/_dynamo/convert_frame.py", line 784, in compile_inner
    return _compile_inner(code, one_graph, hooks, transform)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ryanguo99/pt/pytorch/torch/_dynamo/convert_frame.py", line 933, in _compile_inner
    check_fn = CheckFunctionManager(
               ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ryanguo99/pt/pytorch/torch/_dynamo/guards.py", line 2779, in __init__
    builder, guard_manager = self.build_guards(
                             ^^^^^^^^^^^^^^^^^^
  File "/home/ryanguo99/pt/pytorch/torch/_dynamo/guards.py", line 3037, in build_guards
    guard.create(builder)
  File "/home/ryanguo99/pt/pytorch/torch/_guards.py", line 360, in create
    return self.create_fn(builder, self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ryanguo99/pt/pytorch/torch/_dynamo/guards.py", line 2092, in SHAPE_ENV
    python_code_parts, verbose_code_parts = _get_code_parts(
                                            ^^^^^^^^^^^^^^^^
  File "/home/ryanguo99/pt/pytorch/torch/_dynamo/guards.py", line 2066, in _get_code_parts
    return output_graph.shape_env.produce_guards_verbose(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ryanguo99/pt/pytorch/torch/fx/experimental/symbolic_shapes.py", line 5967, in produce_guards_verbose
    self._check_translation_validate()
  File "/home/ryanguo99/pt/pytorch/torch/fx/experimental/symbolic_shapes.py", line 4211, in _check_translation_validate
    self.validator.validate()
  File "/home/ryanguo99/pt/pytorch/torch/fx/experimental/validator.py", line 569, in validate
    return self._validate()
           ^^^^^^^^^^^^^^^^
  File "/home/ryanguo99/pt/pytorch/torch/fx/experimental/validator.py", line 606, in _validate
    raise ValidationException(
torch.fx.experimental.validator.ValidationException: translation validation failed.

Model:
  ==> L['x'].size()[0]: 2
  ==> L['x'].storage_offset(): 0
  ==> L['x'].stride()[0]: 1
  ==> u0: True

Assertions:
  ==> (== 0 L['x'].storage_offset())
  ==> (== 1 L['x'].stride()[0])
  ==> (== 2 L['x'].size()[0])

Target Expressions:
  ==> (<= 0 (If u0 1 0))
  ==> (== 0 L['x'].storage_offset())
  ==> (== 1 L['x'].stride()[0])
  ==> (== 2 L['x'].size()[0])
  ==> (>= 1 (If u0 1 0))

Failed Source Expressions:
  ==> (Not u0)

Versions

Python 3.12.5, main 9f5153b

cc @chauhang @penguinwu @ezyang @bobrenjc93

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      0