Closed
Description
Platforms: mac, macos
This test was disabled because it is failing in CI. See recent examples and the most recent trunk workflow logs.
Over the past 3 hours, it has been determined flaky in 6 workflow(s) with 12 failures and 6 successes.
Debugging instructions (after clicking on the recent samples link):
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
- Click on the workflow logs linked above
- Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
- Grep for
test_full_dtype - There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.
Sample error message
Traceback (most recent call last):
File "/Users/ec2-user/runner/_work/pytorch/pytorch/test/inductor/test_torchinductor.py", line 12639, in test_full_dtype
ret_opt = fn_opt(pytype, dtype)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 550, in _fn
return fn(*args, **kwargs)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 1364, in __call__
return self._torchdynamo_orig_callable(
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 1155, in __call__
result = self._inner_convert(
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 544, in __call__
return _compile(
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 964, in _compile
guarded_code = compile_inner(code, one_graph, hooks, transform)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 695, in compile_inner
return _compile_inner(code, one_graph, hooks, transform)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_utils_internal.py", line 95, in wrapper_function
return function(*args, **kwargs)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 728, in _compile_inner
out_code = transform_code_object(code, transform)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/bytecode_transformation.py", line 1337, in transform_code_object
transformations(instructions, code_options)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 229, in _fn
return fn(*args, **kwargs)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 657, in transform
tracer.run()
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2887, in run
super().run()
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 1095, in run
while self.step():
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 1007, in step
self.dispatch_table[inst.opcode](self, inst)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 3078, in RETURN_VALUE
self._return(inst)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 3063, in _return
self.output.compile_subgraph(
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 1131, in compile_subgraph
self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 1400, in compile_and_call_fx_graph
compiled_fn = self.call_user_compiler(gm)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 1447, in call_user_compiler
return self._call_user_compiler(gm)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 1496, in _call_user_compiler
raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 1477, in _call_user_compiler
compiled_fn = compiler_fn(gm, self.example_inputs())
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/repro/after_dynamo.py", line 130, in __call__
compiled_gm = compiler_fn(gm, example_inputs)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/backends/inductor.py", line 12, in inductor
return compile_fx(*args, **kwargs)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 1655, in compile_fx
return aot_autograd(
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/backends/common.py", line 72, in __call__
cg = aot_module_simplified(gm, example_inputs, **self.kwargs)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_functorch/aot_autograd.py", line 1087, in aot_module_simplified
compiled_fn = dispatch_and_compile()
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_functorch/aot_autograd.py", line 1063, in dispatch_and_compile
compiled_fn, _ = create_aot_dispatcher_function(
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_functorch/aot_autograd.py", line 524, in create_aot_dispatcher_function
return _create_aot_dispatcher_function(
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_functorch/aot_autograd.py", line 762, in _create_aot_dispatcher_function
compiled_fn, fw_metadata = compiler_fn(
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 196, in aot_dispatch_base
compiled_fw = compiler(fw_module, updated_flat_args)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 1474, in fw_compiler_base
return _fw_compiler_base(model, example_inputs, is_inference)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 1545, in _fw_compiler_base
return inner_compile(
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 548, in compile_fx_inner
return wrap_compiler_debug(_compile_fx_inner, compiler_name="inductor")(
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_dynamo/repro/after_aot.py", line 100, in debug_wrapper
inner_compiled_fn = compiler_fn(gm, example_inputs)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 700, in _compile_fx_inner
compiled_graph = FxGraphCache.load(
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 1475, in load
compiled_graph = compile_fx_fn(
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 611, in codegen_and_compile
compiled_graph = fx_codegen_and_compile(gm, example_inputs, **fx_kwargs)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 921, in fx_codegen_and_compile
compiled_fn = graph.compile_to_fn()
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/graph.py", line 2013, in compile_to_fn
return self.compile_to_module().call
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/graph.py", line 1935, in compile_to_module
return self._compile_to_module()
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/graph.py", line 1967, in _compile_to_module
mod = PyCodeCache.load_by_key_path(
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 3009, in load_by_key_path
mod = _reload_python_module(key, path)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/runtime/compile_tasks.py", line 45, in _reload_python_module
exec(code, mod.__dict__, mod.__dict__)
File "/var/folders/bm/fnn3xd1d39lcpbxrgwys1c140000gn/T/tmpfsdcyzgy/ly/clyn66fxjbs3vshavviwicbdhfpinugbmzots3obrm2zijjmfijc.py", line 50, in <module>
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/async_compile.py", line 302, in wait
scope[key] = result.result()
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 3488, in result
return self.result_fn()
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 2510, in future
result = get_result()
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 2300, in load_fn
future.result()
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/concurrent/futures/_base.py", line 446, in result
return self.__get_result()
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 2341, in _worker_compile_cpp
cpp_builder.build()
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/cpp_builder.py", line 1526, in build
status = run_compile_cmd(build_cmd, cwd=_build_tmp_dir)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/cpp_builder.py", line 350, in run_compile_cmd
return _run_compile_cmd(cmd_line, cwd)
File "/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/_inductor/cpp_builder.py", line 344, in _run_compile_cmd
raise exc.CppCompileError(cmd, output) from e
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
CppCompileError: C++ compile error
Command:
clang++ /var/folders/bm/fnn3xd1d39lcpbxrgwys1c140000gn/T/tmpfsdcyzgy/g3/cg3celq6znuldepm32pw7gahfkbubhbe24fvwshwvms47bvlleoh.cpp -D TORCH_INDUCTOR_CPP_WRAPPER -D STANDALONE_TORCH_HEADER -D C10_USING_CUSTOM_GENERATED_MACROS -D CPU_CAPABILITY_NEON -D AT_BUILD_ARM_VEC256_WITH_SLEEF -shared -fPIC -undefined dynamic_lookup -O3 -DNDEBUG -fno-trapping-math -funsafe-math-optimizations -ffinite-math-only -fno-signed-zeros -fno-math-errno -fno-finite-math-only -fno-unsafe-math-optimizations -ffp-contract=off -Wall -std=c++17 -Wno-unused-variable -Wno-unknown-pragmas -Werror=ignored-optimization-argument -Xclang -fopenmp -I/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/include/python3.9 -I/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/include/python3.9 -I/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/include -I/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/include/TH -I/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/include/THC -I/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/include -I/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/include -I/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/include/TH -I/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/include/THC -D_GLIBCXX_USE_CXX11_ABI=0 -lomp -L/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib -L/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/lib -L/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib -L/Users/ec2-user/runner/_work/_temp/conda_environment_11453819042/lib/python3.9/site-packages/torch/lib -o /var/folders/bm/fnn3xd1d39lcpbxrgwys1c140000gn/T/tmpfsdcyzgy/g3/cg3celq6znuldepm32pw7gahfkbubhbe24fvwshwvms47bvlleoh.so
Output:
Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
You can suppress this exception and fall back to eager by setting:
import torch._dynamo
torch._dynamo.config.suppress_errors = True
To execute this test, run the following from the base repo dir:
python test/inductor/test_torchinductor.py TestFull.test_full_dtype
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Test file path: inductor/test_torchinductor.py
cc @clee2000 @malfet @albanD @ezyang @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @aakhundov