10000 [HIP] Runtime failure for forward conv · Issue #20766 · iree-org/iree · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[HIP] Runtime failure for forward conv #20766

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zjgarvey opened this issue May 9, 2025 · 2 comments · Fixed by #20787
Closed

[HIP] Runtime failure for forward conv #20766

zjgarvey opened this issue May 9, 2025 · 2 comments · Fixed by #20787
Labels
bug 🐞 Something isn't working

Comments

@zjgarvey
Copy link
Contributor
zjgarvey commented May 9, 2025 •
edited
Loading

What happened?

The following IR compiles successfully for gfx942, but results in an error when running:

Error

EXEC @conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g
c/runtime/src/iree/hal/drivers/hip/event_semaphore.c:786: ABORTED; the semaphore was aborted; while invoking native function hal.fence.await; while calling import; 
[ 0] bytecode module.conv_2d_float

Reproducer IR

#map = affine_map<(d0, d1, d2, d3, d4, d5, d6) -> (d0, d1 + d4, d2 + d5, d6)>
#map1 = affine_map<(d0, d1, d2, d3, d4, d5, d6) -> (d3, d4, d5, d6)>
#map2 = affine_map<(d0, d1, d2, d3, d4, d5, d6) -> (d0, d1, d2, d3)>
module @module {
  util.func public @conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async(%arg0: !hal.buffer_view, %arg1: !hal.buffer_view, %arg2: !hal.fence, %arg3: !hal.fence) -> !hal.buffer_view attributes {inlining_policy = #util.inline.never, iree.abi.model = "coarse-fences", iree.abi.stub, preprocessing_pipeline = #util.preprocessing_pipeline<"iree-preprocessing-make-single-dispatch">} {
    %cst = arith.constant 0.000000e+00 : f32
    %0 = hal.tensor.import wait(%arg2) => %arg0 : !hal.buffer_view -> tensor<11x470x725x32xf32>
    %1 = hal.tensor.import wait(%arg2) => %arg1 : !hal.buffer_view -> tensor<224x1x1x32xf32>
    %2 = tensor.empty() : tensor<11x470x725x224xf32>
    %3 = linalg.fill ins(%cst : f32) outs(%2 : tensor<11x470x725x224xf32>) -> tensor<11x470x725x224xf32>
    %4 = linalg.generic {indexing_maps = [#map, #map1, #map2], iterator_types = ["parallel", "parallel", "parallel", "parallel", "reduction", "reduction", "reduction"]} ins(%0, %1 : tensor<11x470x725x32xf32>, tensor<224x1x1x32xf32>) outs(%3 : tensor<11x470x725x224xf32>) {
    ^bb0(%in: f32, %in_0: f32, %out: f32):
      %7 = arith.mulf %in, %in_0 : f32
      %8 = arith.addf %out, %7 : f32
      linalg.yield %8 : f32
    } -> tensor<11x470x725x224xf32>
    %5 = hal.tensor.barrier join(%4 : tensor<11x470x725x224xf32>) => %arg3 : !hal.fence
    %6 = hal.tensor.export %5 : tensor<11x470x725x224xf32> -> !hal.buffer_view
    util.return %6 : !hal.buffer_view
  }
  util.func public @conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g(%arg0: !hal.buffer_view, %arg1: !hal.buffer_view) -> !hal.buffer_view attributes {iree.abi.stub} {
    %0 = util.null : !hal.fence
    %c-1_i32 = arith.constant -1 : i32
    %c0 = arith.constant 0 : index
    %device_0 = hal.devices.get %c0 : !hal.device
    %fence = hal.fence.create device(%device_0 : !hal.device) flags("None") : !hal.fence
    %1 = util.call @conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async(%arg0, %arg1, %0, %fence) : (!hal.buffer_view, !hal.buffer_view, !hal.fence, !hal.fence) -> !hal.buffer_view
    %status = hal.fence.await until([%fence]) timeout_millis(%c-1_i32) flags("None") : i32
    util.return %1 : !hal.buffer_view
  }
}

Steps to reproduce your issue

  1. Compile the IR with iree-compile --iree-hal-target-backends=rocm --iree-hip-target=gfx942 -o repro.vmfb
  2. Run with iree-run-module --module=<repro.vmfb> --function=conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g --device=hip --input="11x470x725x32xf32=1.0" --input="224x1x1x32xf32=1.0"

What component(s) does this issue relate to?

Runtime

Version information

local build @ efeb94b

Additional context

Running with --trace_execution gives:

[module.__init+00000000]    <block>
[module.__init+00000001]    %i0 = vm.const.i32 1  // 0x00000001
[module.__init+00000008]    %r0 = vm.const.ref.zero
[module.__init+0000000B]    %i1 = vm.const.i32 14  // 0x0000000E
[module.__init+00000012]    %i2:3 = vm.const.i64 -1  // 0xFFFFFFFFFFFFFFFF
[module.__init+0000001D]    %i4 = vm.const.i32 18  // 0x00000012
[module.__init+00000024]    %i5 = vm.const.i32.zero
[module.__init+00000027]    %i6:7 = vm.const.i64.zero
[module.__init+0000002A]    %i8:9 = vm.const.i64 1  // 0x0000000000000001
[module.__init+00000035]    %r1 = vm.const.ref.zero
[module.__init+00000038]    %i10 = vm.call @hal.devices.count()
[module.__init+00000044]    %i10:11 = vm.ext.i32.i64.s %i10(1)
[module.__init+00000049]    vm.br ^00000064(%r1(null)->%r2, %i6(0)->%i12, %i7(0)->%i13, %i6(0)->%i14, %i7(0)->%i15)
[module.__init+00000065]    %i16 = vm.cmp.nz.ref %r2(null)
[module.__init+0000006A]    %i16 = vm.xor.i32 %i16(0), %i0(1)
[module.__init+00000071]    %i17 = vm.cmp.lt.i64.s %i12:13(0), %i10:11(1)
[module.__init+00000078]    %i17 = vm.and.i32 %i16(1), %i17(1)
[module.__init+0000007F]    vm.cond_br %i17(1), ^0000008E(), ^00000158()
[module.__init+0000008F]    %i16 = vm.trunc.i64.i32 %i12:13(0)
[module.__init+00000094]    %r2 = vm.call @hal.devices.get(%i16(0))
[module.__init+000000A2]    %r3 = vm.const.ref.rodata 0  // 0x0x5cf05d77fe5c 13b
[module.__init+000000A9]    %r4 = vm.const.ref.rodata 1  // 0x0x5cf05d77fe78 3b
[module.__init+000000B0]    %i16, %i18 = vm.call @hal.device.query.i64(%r2(!hal.device/0x0x5cf05e379950), %r3(!vm.buffer/0x0x5cf05d77dc90), %r4(!vm.buffer/0x0x5cf05d77dcb8))
[module.__init+000000C4]    %i17 = vm.cmp.nz.i64 %i18:19(1)
[module.__init+000000C9]    %i16 = vm.select.i32 %i16(1) ? %i17(1) : %i5(0)
[module.__init+000000D2]    vm.cond_br %i16(1), ^000000E6(), ^0000011E(%i5(0)->%i16)
[module.__init+000000E7]    %r3 = vm.const.ref.rodata 2  // 0x0x5cf05d77fe88 21b
[module.__init+000000EE]    %r4 = vm.const.ref.rodata 3  // 0x0x5cf05d77feac 13b
[module.__init+000000F5]    %i16, %i18 = vm.call @hal.device.query.i64(%r2(!hal.device/0x0x5cf05e379950), %r3(!vm.buffer/0x0x5cf05d77dce0), %r4(!vm.buffer/0x0x5cf05d77dd08))
[module.__init+00000108]    %i17 = vm.cmp.nz.i64 %i18:19(1)
[module.__init+0000010D]    %i16 = vm.select.i32 %i16(1) ? %i17(1) : %i5(0)
[module.__init+00000116]    vm.br ^0000011E()
[module.__init+0000011F]    %i17 = vm.cmp.eq.i64 %i14:15(0), %i6:7(0)
[module.__init+00000126]    %i18:19 = vm.select.i64 %i16(1) ? %i8:9(1) : %i6:7(0)
[module.__init+0000012F]    %i14:15 = vm.add.i64 %i14:15(0), %i18:19(1)
[module.__init+00000136]    %i16 = vm.and.i32 %i16(1), %i17(1)
[module.__init+0000013D]    %r2 = vm.select.ref %i16(1) ? %r2(!hal.device/0x0x5cf05e379950) : %r1(null) -> !hal.device
[module.__init+0000014A]    %i12:13 = vm.add.i64 %i12:13(0), %i8:9(1)
[module.__init+00000151]    vm.br ^00000064()
[module.__init+00000065]    %i16 = vm.cmp.nz.ref %r2(!hal.device/0x0x5cf05e379950)
[module.__init+0000006A]    %i16 = vm.xor.i32 %i16(1), %i0(1)
[module.__init+00000071]    %i17 = vm.cmp.lt.i64.s %i12:13(1), %i10:11(1)
[module.__init+00000078]    %i17 = vm.and.i32 %i16(0), %i17(0)
[module.__init+0000007F]    vm.cond_br %i17(0), ^0000008E(), ^00000158()
[module.__init+00000159]    vm.cond_br %i16(0), ^00000168(), ^000005F0()
[module.__init+000005F1]    %r1 = vm.const.ref.rodata 2  // 0x0x5cf05d77fe88 21b
[module.__init+000005F8]    %r3 = vm.const.ref.rodata 3  // 0x0x5cf05d77feac 13b
[module.__init+000005FF]    %i0, %i8 = vm.call @hal.device.query.i64(%r2(!hal.device/0x0x5cf05e379950), %r1(!vm.buffer/0x0x5cf05d77dce0), %r3(!vm.buffer/0x0x5cf05d77dd08))
[module.__init+00000612]    %i4 = vm.cmp.nz.i64 %i8:9(1)
[module.__init+00000617]    %i0 = vm.select.i32 %i0(1) ? %i4(1) : %i5(0)
[module.__init+00000620]    %i4:5 = vm.select.i64 %i0(1) ? %i6:7(0) : %i2:3(-1)
[module.__init+00000629]    %i0 = vm.cmp.eq.i64 %i4:5(0), %i6:7(0)
[module.__init+00000630]    vm.global.store.ref %r2(!hal.device/0x0x5cf05e379950), .refs[0] : !hal.device
[module.__init+0000063B]    vm.cond_br %i0(1), ^0000064A(), ^0000068C()
[module.__init+0000064B]    %r1 = vm.const.ref.rodata 4  // 0x0x5cf05d780b30 7480b
[module.__init+00000652]    %r0 = vm.call @hal.executable.create(%r2(!hal.device/0x0x5cf05e379950), %i2(4294967295), %r3(!vm.buffer/0x0x5cf05d77dd08), %r1(!vm.buffer/0x0x5cf05d77dd30), %r0(null))
[module.__init+00000668]    vm.global.store.ref %r0(!hal.executable/0x0x5cf05fd6b780), .refs[1] : !hal.executable
[module.__init+00000673]    %r0 = vm.call @module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply()
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+00000000]    <block>
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+00000001]    %i0 = vm.const.i32 13  // 0x0000000D
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+00000008]    %i1 = vm.const.i32 28  // 0x0000001C
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+0000000F]    %i2 = vm.const.i32 2  // 0x00000002
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+00000016]    %r0 = vm.const.ref.zero
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+00000019]    %i3 = vm.const.i32 1  // 0x00000001
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+00000020]    %i4 = vm.const.i32 117133  // 0x0001C98D
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+00000027]    %i5 = vm.const.i32 7  // 0x00000007
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+0000002E]    %i6 = vm.const.i32 3  // 0x00000003
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+00000035]    %i7 = vm.const.i32.zero
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+00000038]    %i8:9 = vm.const.i64 3358432000  // 0x00000000C82D9B00
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+00000043]    %i10:11 = vm.const.i64 28672  // 0x0000000000007000
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+0000004E]    %i12:13 = vm.const.i64 479776000  // 0x000000001C98CD00
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+00000059]    %i14:15 = vm.const.i64.zero
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+0000005C]    %i16:17 = vm.const.i64 -1  // 0xFFFFFFFFFFFFFFFF
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+00000067]    %r1 = vm.global.load.ref .refs[0](!hal.device/0x0x5cf05e379950) : !hal.device
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+00000072]    %r2 = vm.global.load.ref .refs[1](!hal.executable/0x0x5cf05fd6b780) : !hal.executable
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+0000007D]    %r1 = vm.call @hal.command_buffer.create(%r1(!hal.device/0x0x5cf05e379950), %i7(0), %i6(3), %i16(4294967295), %i6(3))
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+00000092]    vm.call.varadic @hal.command_buffer.dispatch(%r1(!hal.command_buffer/0x0x5cf05fd6e5b0), %r2(!hal.executable/0x0x5cf05fd6b780), %i7(0), %i5(7), %i4(117133), %i3(1), %i14(0), %i7(0), %i7(0), %r0(null), %i14(0), %i12(479776000), %i7(0), %i3(1), %r0(null), %i14(0), %i10(28672), %i7(0), %i2(2), %r0(null), %i14(0), %i8(3358432000))
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+000000DC]    vm.call @hal.command_buffer.execution_barrier(%r1(!hal.command_buffer/0x0x5cf05fd6e5b0), %i1(28), %i0(13), %i14(0))
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+000000EE]    vm.call @hal.command_buffer.finalize(%r1(!hal.command_buffer/0x0x5cf05fd6e5b0))
[module.__conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_memoize_apply+000000FA]    vm.return %r1(!hal.command_buffer/0x0x5cf05fd6e5b0)
[module.__init+0000067E]    vm.global.store.ref %r0(!hal.command_buffer/0x0x5cf05fd6e5b0), .refs[2] : !hal.command_buffer
[module.__init+00000689]    vm.return 
EXEC @conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g+00000000]    <block>
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g+00000001]    %r2 = vm.const.ref.zero
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g+00000004]    %i0 = vm.const.i32 -1  // 0xFFFFFFFF
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g+0000000B]    %i2:3 = vm.const.i64.zero
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g+0000000E]    %i1 = vm.const.i32.zero
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g+00000011]    %r3 = vm.call @hal.devices.get(%i1(0))
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g+0000001E]    %r3 = vm.call @hal.fence.create(%r3(!hal.device/0x0x5cf05e379950), %i2(0))
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g+0000002E]    %r0 = vm.call @module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async(%r0(!hal.buffer_view/0x0x5cf05fd6eeb0), %r1(!hal.buffer_view/0x0x5cf05fda00c0), %r2(null), %r3(!hal.fence/0x0x5cf05fd99610))
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+00000000]    <block>
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+00000001]    %i0 = vm.const.i32 48  // 0x00000030
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+00000008]    %i1 = vm.const.i32 3075  // 0x00000C03
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+0000000F]    %i2 = vm.const.i32 16  // 0x00000010
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+00000016]    %i3 = vm.const.i32 1  // 0x00000001
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+0000001D]    %i4 = vm.const.i32 553648160  // 0x21000020
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+00000024]    %i6:7 = vm.const.i64 11  // 0x000000000000000B
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+0000002F]    %i8:9 = vm.const.i64 470  // 0x00000000000001D6
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+0000003A]    %i10:11 = vm.const.i64 725  // 0x00000000000002D5
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+00000045]    %i12:13 = vm.const.i64 32  // 0x0000000000000020
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+00000050]    %i14:15 = vm.const.i64 224  // 0x00000000000000E0
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+0000005B]    %i16:17 = vm.const.i64 1  // 0x0000000000000001
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+00000066]    %i18:19 = vm.const.i64 479776000  // 0x000000001C98CD00
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+00000071]    %i20:21 = vm.const.i64 28672  // 0x0000000000007000
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+0000007C]    %i22:23 = vm.const.i64 3358432000  // 0x00000000C82D9B00
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+00000087]    %i24:25 = vm.const.i64.zero
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+0000008A]    %i26:27 = vm.const.i64 -1  // 0xFFFFFFFFFFFFFFFF
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+00000095]    %r4 = vm.global.load.ref .refs[0](!hal.device/0x0x5cf05e379950) : !hal.device
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+000000A0]    %r5 = vm.global.load.ref .refs[2](!hal.command_buffer/0x0x5cf05fd6e5b0) : !hal.command_buffer
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+000000AB]    %r6 = vm.const.ref.rodata 5  // 0x0x5cf05d77fee0 6b
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+000000B2]    vm.call.varadic @hal.buffer_view.assert(%r0(!hal.buffer_view/0x0x5cf05fd6eeb0), %r6(!vm.buffer/0x0x5cf05d77dd58), %i4(553648160), %i3(1), %i6(11), %i8(470), %i10(725), %i12(32))
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+000000D8]    %r0 = vm.call @hal.buffer_view.buffer(%r0(!hal.buffer_view/0x0x5cf05fd6eeb0))
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+000000E6]    %r7 = vm.call @hal.device.allocator(%r4(!hal.device/0x0x5cf05e379950))
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+000000F4]    vm.call @hal.buffer.assert(%r0(!hal.buffer/0x0x5cf05fd6e6c0), %r6(!vm.buffer/0x0x5cf05d77dd58), %r7(!hal.allocator/0x0x5cf05fd66220), %i18(479776000), %i2(16), %i1(3075))
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+0000010A]    vm.call.varadic @hal.buffer_view.assert(%r1(!hal.buffer_view/0x0x5cf05fda00c0), %r6(!vm.buffer/0x0x5cf05d77dd58), %i4(553648160), %i3(1), %i14(224), %i16(1), %i16(1), %i12(32))
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+00000130]    %r1 = vm.call @hal.buffer_view.buffer(%r1(!hal.buffer_view/0x0x5cf05fda00c0))
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+0000013E]    vm.call @hal.buffer.assert(%r1(!hal.buffer/0x0x5cf05fd6ef10), %r6(!vm.buffer/0x0x5cf05d77dd58), %r7(!hal.allocator/0x0x5cf05fd66220), %i20(28672), %i2(16), %i1(3075))
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+00000154]    %r6 = vm.call @hal.fence.create(%r4(!hal.device/0x0x5cf05e379950), %i24(0))
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+00000164]    %r2 = vm.call @hal.device.queue.alloca(%r4(!hal.device/0x0x5cf05e379950), %i26(4294967295), %r2(null), %r6(!hal.fence/0x0x5cf05fd99ac0), %i24(0), %i0(48), %i1(3075), %i22(3358432000), %i24(0))
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+00000182]    vm.call.varadic @hal.device.queue.execute.indirect(%r4(!hal.device/0x0x5cf05e379950), %i
8000
26(4294967295), %r6(!hal.fence/0x0x5cf05fd99ac0), %r3(!hal.fence/0x0x5cf05fd99610), %r5(!hal.command_buffer/0x0x5cf05fd6e5b0), %i24(0), %r0(!hal.buffer/0x0x5cf05fd6e6c0), %i24(0), %i18(479776000), %r1(!hal.buffer/0x0x5cf05fd6ef10), %i24(0), %i20(28672), %r2(!hal.buffer/0x0x5cf05fd99b00), %i24(0), %i22(3358432000))
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+000001BA]    %r0 = vm.call.varadic @hal.buffer_view.create(%r2(!hal.buffer/0x0x5cf05fd99b00), %i24(0), %i22(3358432000), %i4(553648160), %i3(1), %i6(11), %i8(470), %i10(725), %i14(224))
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async+000001E6]    vm.return %r0(!hal.buffer_view/0x0x5cf05fd99d60)
[module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g+00000042]    %i0 = vm.call.varadic @hal.fence.await(%i0(4294967295), %i2(0), %r3(!hal.fence/0x0x5cf05fd99610))
iree/runtime/src/iree/hal/drivers/hip/event_semaphore.c:786: ABORTED; the semaphore was aborted; while invoking native function hal.fence.await; while calling import; 
[ 0] bytecode module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g:92 repro.mlir:29:5; invoking function 'conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g'
@zjgarvey zjgarvey added the bug 🐞 Something isn't working label May 9, 2025
@zjgarvey
Copy link
Contributor Author
zjgarvey commented May 9, 2025

Here is the printout with AMD_SERIALIZE_KERNEL=3 AMD_LOG_LEVEL=4 :

:3:rocdevice.cpp            :465 : 159550074485 us: [pid:740193 tid: 0x778c75a53740] Initializing HSA stack.
:3:rocdevice.cpp            :551 : 159550343568 us: [pid:740193 tid: 0x778c75a53740] Enumerated GPU agents = 8
:3:rocdevice.cpp            :230 : 159550343726 us: [pid:740193 tid: 0x778c75a53740] Numa selects cpu agent[0]=0x7ce1550(fine=0x7d15d90,coarse=0x7d1f340) for gpu agent=0x7d4b120 CPU<->GPU XGMI=0
:3:rocsettings.cpp          :282 : 159550343733 us: [pid:740193 tid: 0x778c75a53740] Using dev kernel arg wa = 2
:3:comgrctx.cpp             :33  : 159550343739 us: [pid:740193 tid: 0x778c75a53740] Loading COMGR library.
:3:comgrctx.cpp             :128 : 159550348705 us: [pid:740193 tid: 0x778c75a53740] Loaded COMGR library version 3.0.
:3:rocdevice.cpp            :1785: 159550349111 us: [pid:740193 tid: 0x778c75a53740] Gfx Major/Minor/Stepping: 9/4/2
:3:rocdevice.cpp            :1787: 159550349122 us: [pid:740193 tid: 0x778c75a53740] HMM support: 1, XNACK: 0, Direct host access: 0
:3:rocdevice.cpp            :1789: 159550349126 us: [pid:740193 tid: 0x778c75a53740] Max SDMA Read Mask: 0xffff, Max SDMA Write Mask: 0xffff
:4:rocdevice.cpp            :2210: 159550349535 us: [pid:740193 tid: 0x778c75a53740] Allocate hsa host memory 0x77843e600000, size 0x100000, numa_node = 0
:3:rocdevice.cpp            :230 : 159550349858 us: [pid:740193 tid: 0x778c75a53740] Numa selects cpu agent[0]=0x7ce1550(fine=0x7d15d90,coarse=0x7d1f340) for gpu agent=0x8381880 CPU<->GPU XGMI=0
:3:rocsettings.cpp          :282 : 159550349865 us: [pid:740193 tid: 0x778c75a53740] Using dev kernel arg wa = 2
:3:rocdevice.cpp            :1785: 159550350124 us: [pid:740193 tid: 0x778c75a53740] Gfx Major/Minor/Stepping: 9/4/2
:3:rocdevice.cpp            :1787: 159550350134 us: [pid:740193 tid: 0x778c75a53740] HMM support: 1, XNACK: 0, Direct host access: 0
:3:rocdevice.cpp            :1789: 159550350138 us: [pid:740193 tid: 0x778c75a53740] Max SDMA Read Mask: 0xffff, Max SDMA Write Mask: 0xffff
:4:rocdevice.cpp            :2210: 159550350606 us: [pid:740193 tid: 0x778c75a53740] Allocate hsa host memory 0x77843e400000, size 0x100000, numa_node = 0
:3:rocdevice.cpp            :230 : 159550350817 us: [pid:740193 tid: 0x778c75a53740] Numa selects cpu agent[0]=0x7ce1550(fine=0x7d15d90,coarse=0x7d1f340) for gpu agent=0x842ceb0 CPU<->GPU XGMI=0
:3:rocsettings.cpp          :282 : 159550350823 us: [pid:740193 tid: 0x778c75a53740] Using dev kernel arg wa = 2
:3:rocdevice.cpp            :1785: 159550351063 us: [pid:740193 tid: 0x778c75a53740] Gfx Major/Minor/Stepping: 9/4/2
:3:rocdevice.cpp            :1787: 159550351073 us: [pid:740193 tid: 0x778c75a53740] HMM support: 1, XNACK: 0, Direct host access: 0
:3:rocdevice.cpp            :1789: 159550351077 us: [pid:740193 tid: 0x778c75a53740] Max SDMA Read Mask: 0xffff, Max SDMA Write Mask: 0xffff
:4:rocdevice.cpp            :2210: 159550351568 us: [pid:740193 tid: 0x778c75a53740] Allocate hsa host memory 0x77843e200000, size 0x100000, numa_node = 0
:3:rocdevice.cpp            :230 : 159550351776 us: [pid:740193 tid: 0x778c75a53740] Numa selects cpu agent[0]=0x7ce1550(fine=0x7d15d90,coarse=0x7d1f340) for gpu agent=0x84d8530 CPU<->GPU XGMI=0
:3:rocsettings.cpp          :282 : 159550351782 us: [pid:740193 tid: 0x778c75a53740] Using dev kernel arg wa = 2
:3:rocdevice.cpp            :1785: 159550352017 us: [pid:740193 tid: 0x778c75a53740] Gfx Major/Minor/Stepping: 9/4/2
:3:rocdevice.cpp            :1787: 159550352028 us: [pid:740193 tid: 0x778c75a53740] HMM support: 1, XNACK: 0, Direct host access: 0
:3:rocdevice.cpp            :1789: 159550352031 us: [pid:740193 tid: 0x778c75a53740] Max SDMA Read Mask: 0xffff, Max SDMA Write Mask: 0xffff
:4:rocdevice.cpp            :2210: 159550352396 us: [pid:740193 tid: 0x778c75a53740] Allocate hsa host memory 0x77843e000000, size 0x100000, numa_node = 0
:3:rocdevice.cpp            :230 : 159550352638 us: [pid:740193 tid: 0x778c75a53740] Numa selects cpu agent[1]=0x7d48270(fine=0x7d484b0,coarse=0x7d49020) for gpu agent=0x8583bc0 CPU<->GPU XGMI=0
:3:rocsettings.cpp          :282 : 159550352645 us: [pid:740193 tid: 0x778c75a53740] Using dev kernel arg wa = 2
:3:rocdevice.cpp            :1785: 159550352881 us: [pid:740193 tid: 0x778c75a53740] Gfx Major/Minor/Stepping: 9/4/2
:3:rocdevice.cpp            :1787: 159550352889 us: [pid:740193 tid: 0x778c75a53740] HMM support: 1, XNACK: 0, Direct host access: 0
:3:rocdevice.cpp            :1789: 159550352892 us: [pid:740193 tid: 0x778c75a53740] Max SDMA Read Mask: 0xffff, Max SDMA Write Mask: 0xffff
:4:rocdevice.cpp            :2210: 159550353228 us: [pid:740193 tid: 0x778c75a53740] Allocate hsa host memory 0x77843de00000, size 0x100000, numa_node = 1
:3:rocdevice.cpp            :230 : 159550353472 us: [pid:740193 tid: 0x778c75a53740] Numa selects cpu agent[1]=0x7d48270(fine=0x7d484b0,coarse=0x7d49020) for gpu agent=0x862f290 CPU<->GPU XGMI=0
:3:rocsettings.cpp          :282 : 159550353479 us: [pid:740193 tid: 0x778c75a53740] Using dev kernel arg wa = 2
:3:rocdevice.cpp            :1785: 159550353702 us: [pid:740193 tid: 0x778c75a53740] Gfx Major/Minor/Stepping: 9/4/2
:3:rocdevice.cpp            :1787: 159550353710 us: [pid:740193 tid: 0x778c75a53740] HMM support: 1, XNACK: 0, Direct host access: 0
:3:rocdevice.cpp            :1789: 159550353713 us: [pid:740193 tid: 0x778c75a53740] Max SDMA Read Mask: 0xffff, Max SDMA Write Mask: 0xffff
:4:rocdevice.cpp            :2210: 159550354045 us: [pid:740193 tid: 0x778c75a53740] Allocate hsa host memory 0x77843dc00000, size 0x100000, numa_node = 1
:3:rocdevice.cpp            :230 : 159550354292 us: [pid:740193 tid: 0x778c75a53740] Numa selects cpu agent[1]=0x7d48270(fine=0x7d484b0,coarse=0x7d49020) for gpu agent=0x86da8f0 CPU<->GPU XGMI=0
:3:rocsettings.cpp          :282 : 159550354298 us: [pid:740193 tid: 0x778c75a53740] Using dev kernel arg wa = 2
:3:rocdevice.cpp            :1785: 159550354520 us: [pid:740193 tid: 0x778c75a53740] Gfx Major/Minor/Stepping: 9/4/2
:3:rocdevice.cpp            :1787: 159550354528 us: [pid:740193 tid: 0x778c75a53740] HMM support: 1, XNACK: 0, Direct host access: 0
:3:rocdevice.cpp            :1789: 159550354531 us: [pid:740193 tid: 0x778c75a53740] Max SDMA Read Mask: 0xffff, Max SDMA Write Mask: 0xffff
:4:rocdevice.cpp            :2210: 159550354870 us: [pid:740193 tid: 0x778c75a53740] Allocate hsa host memory 0x77843da00000, size 0x100000, numa_node = 1
:3:rocdevice.cpp            :230 : 159550355125 us: [pid:740193 tid: 0x778c75a53740] Numa selects cpu agent[1]=0x7d48270(fine=0x7d484b0,coarse=0x7d49020) for gpu agent=0x8785f70 CPU<->GPU XGMI=0
:3:rocsettings.cpp          :282 : 159550355134 us: [pid:740193 tid: 0x778c75a53740] Using dev kernel arg wa = 2
:3:rocdevice.cpp            :1785: 159550355356 us: [pid:740193 tid: 0x778c75a53740] Gfx Major/Minor/Stepping: 9/4/2
:3:rocdevice.cpp            :1787: 159550355364 us: [pid:740193 tid: 0x778c75a53740] HMM support: 1, XNACK: 0, Direct host access: 0
:3:rocdevice.cpp            :1789: 159550355367 us: [pid:740193 tid: 0x778c75a53740] Max SDMA Read Mask: 0xffff, Max SDMA Write Mask: 0xffff
:4:rocdevice.cpp            :2210: 159550355755 us: [pid:740193 tid: 0x778c75a53740] Allocate hsa host memory 0x77843d800000, size 0x100000, numa_node = 1
:4:rocdevice.cpp            :2210: 159550355925 us: [pid:740193 tid: 0x778c75a53740] Allocate hsa host memory 0x778c75724000, size 0x1c0, numa_node = 0
:3:hip_context.cpp          :54  : 159550355965 us: [pid:740193 tid: 0x778c75a53740] Direct Dispatch: 1
:3:hip_context.cpp          :164 : 159550356410 us: [pid:740193 tid: 0x778c75a53740] �[32m hipInit ( 0 ) �[0m
:3:hip_context.cpp          :170 : 159550356422 us: [pid:740193 tid: 0x778c75a53740] hipInit: Returned hipSuccess : 
:3:hip_device_runtime.cpp   :673 : 159550356445 us: [pid:740193 tid: 0x778c75a53740] �[32m hipGetDeviceCount ( 0x7ffd88c124c4 ) �[0m
:3:hip_device_runtime.cpp   :675 : 159550356449 us: [pid:740193 tid: 0x778c75a53740] hipGetDeviceCount: Returned hipSuccess : 
:3:hip_device.cpp           :351 : 159550356454 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGet ( 0x7ffd88c124ec, 0 ) �[0m
:3:hip_device.cpp           :353 : 159550356457 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGet: Returned hipSuccess : 
:3:hip_device.cpp           :416 : 159550356468 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetName ( 0x7ffd88c12590, 128, 0 ) �[0m
:3:hip_device.cpp           :436 : 159550356472 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetName: Returned h
10000
ipSuccess : 
:3:hip_device.cpp           :440 : 159550356475 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetUuid ( 0x7ffd88c124c8, 0 ) �[0m
:3:hip_device.cpp           :454 : 159550356478 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetUuid: Returned hipSuccess : 
:3:hip_device.cpp           :351 : 159550356483 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGet ( 0x7ffd88c124ec, 1 ) �[0m
:3:hip_device.cpp           :353 : 159550356486 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGet: Returned hipSuccess : 
:3:hip_device.cpp           :416 : 159550356489 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetName ( 0x7ffd88c12590, 128, 1 ) �[0m
:3:hip_device.cpp           :436 : 159550356493 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetName: Returned hipSuccess : 
:3:hip_device.cpp           :440 : 159550356496 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetUuid ( 0x7ffd88c124c8, 1 ) �[0m
:3:hip_device.cpp           :454 : 159550356499 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetUuid: Returned hipSuccess : 
:3:hip_device.cpp           :351 : 159550356503 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGet ( 0x7ffd88c124ec, 2 ) �[0m
:3:hip_device.cpp           :353 : 159550356506 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGet: Returned hipSuccess : 
:3:hip_device.cpp           :416 : 159550356509 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetName ( 0x7ffd88c12590, 128, 2 ) �[0m
:3:hip_device.cpp           :436 : 159550356512 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetName: Returned hipSuccess : 
:3:hip_device.cpp           :440 : 159550356515 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetUuid ( 0x7ffd88c124c8, 2 ) �[0m
:3:hip_device.cpp           :454 : 159550356518 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetUuid: Returned hipSuccess : 
:3:hip_device.cpp           :351 : 159550356522 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGet ( 0x7ffd88c124ec, 3 ) �[0m
:3:hip_device.cpp           :353 : 159550356525 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGet: Returned hipSuccess : 
:3:hip_device.cpp           :416 : 159550356529 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetName ( 0x7ffd88c12590, 128, 3 ) �[0m
:3:hip_device.cpp           :436 : 159550356532 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetName: Returned hipSuccess : 
:3:hip_device.cpp           :440 : 159550356535 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetUuid ( 0x7ffd88c124c8, 3 ) �[0m
:3:hip_device.cpp           :454 : 159550356537 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetUuid: Returned hipSuccess : 
:3:hip_device.cpp           :351 : 159550356541 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGet ( 0x7ffd88c124ec, 4 ) �[0m
:3:hip_device.cpp           :353 : 159550356544 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGet: Returned hipSuccess : 
:3:hip_device.cpp           :416 : 159550356547 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetName ( 0x7ffd88c12590, 128, 4 ) �[0m
:3:hip_device.cpp           :436 : 159550356551 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetName: Returned hipSuccess : 
:3:hip_device.cpp           :440 : 159550356554 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetUuid ( 0x7ffd88c124c8, 4 ) �[0m
:3:hip_device.cpp           :454 : 159550356557 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetUuid: Returned hipSuccess : 
:3:hip_device.cpp           :351 : 159550356560 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGet ( 0x7ffd88c124ec, 5 ) �[0m
:3:hip_device.cpp           :353 : 159550356563 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGet: Returned hipSuccess : 
:3:hip_device.cpp           :416 : 159550356566 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetName ( 0x7ffd88c12590, 128, 5 ) �[0m
:3:hip_device.cpp           :436 : 159550356572 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetName: Returned hipSuccess : 
:3:hip_device.cpp           :440 : 159550356575 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetUuid ( 0x7ffd88c124c8, 5 ) �[0m
:3:hip_device.cpp           :454 : 159550356579 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetUuid: Returned hipSuccess : 
:3:hip_device.cpp           :351 : 159550356583 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGet ( 0x7ffd88c124ec, 6 ) �[0m
:3:hip_device.cpp           :353 : 159550356586 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGet: Returned hipSuccess : 
:3:hip_device.cpp           :416 : 159550356589 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetName ( 0x7ffd88c12590, 128, 6 ) �[0m
:3:hip_device.cpp           :436 : 159550356592 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetName: Returned hipSuccess : 
:3:hip_device.cpp           :440 : 159550356595 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetUuid ( 0x7ffd88c124c8, 6 ) �[0m
:3:hip_device.cpp           :454 : 159550356598 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetUuid: Returned hipSuccess : 
:3:hip_device.cpp           :351 : 159550356602 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGet ( 0x7ffd88c124ec, 7 ) �[0m
:3:hip_device.cpp           :353 : 159550356604 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGet: Returned hipSuccess : 
:3:hip_device.cpp           :416 : 159550356608 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetName ( 0x7ffd88c12590, 128, 7 ) �[0m
:3:hip_device.cpp           :436 : 159550356612 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetName: Returned hipSuccess : 
:3:hip_device.cpp           :440 : 159550356615 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetUuid ( 0x7ffd88c124c8, 7 ) �[0m
:3:hip_device.cpp           :454 : 159550356618 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetUuid: Returned hipSuccess : 
:3:hip_context.cpp          :400 : 159550356622 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDevicePrimaryCtxRetain ( 0x88d4b60, 0 ) �[0m
:3:hip_context.cpp          :411 : 159550356626 us: [pid:740193 tid: 0x778c75a53740] hipDevicePrimaryCtxRetain: Returned hipSuccess : 
:3:hip_context.cpp          :191 : 159550356629 us: [pid:740193 tid: 0x778c75a53740] �[32m hipCtxSetCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :205 : 159550356632 us: [pid:740193 tid: 0x778c75a53740] hipCtxSetCurrent: Returned hipSuccess : 
:3:hip_stream.cpp           :268 : 159550356640 us: [pid:740193 tid: 0x778c75a53740] �[32m hipStreamCreateWithFlags ( 0x88d4b70, 1 ) �[0m
:3:rocdevice.cpp            :3036: 159550356650 us: [pid:740193 tid: 0x778c75a53740] Number of allocated hardware queues with low priority: 0, with normal priority: 0, with high priority: 0, maximum per priority is: 4
:3:rocdevice.cpp            :3114: 159550402827 us: [pid:740193 tid: 0x778c75a53740] Created SWq=0x778c75716000 to map on HWq=0x77843ce00000 with size 16384 with priority 1, cooperative: 0
:3:rocdevice.cpp            :3207: 159550402861 us: [pid:740193 tid: 0x778c75a53740] acquireQueue refCount: 0x77843ce00000 (1)
:4:rocdevice.cpp            :2371: 159550402893 us: [pid:740193 tid: 0x778c75a53740] Allocate hsa device memory 0x77843ca00000, size 0x100000
:3:devprogram.cpp           :2658: 159550576582 us: [pid:740193 tid: 0x778c75a53740] Using Code Object V5.
:4:rocdevice.cpp            :2210: 159550578240 us: [pid:740193 tid: 0x778c75a53740] Allocate hsa host memory 0x77843c400000, size 0x400000, numa_node = 0
:3:hip_stream.cpp           :274 : 159550578308 us: [pid:740193 tid: 0x778c75a53740] hipStreamCreateWithFlags: Returned hipSuccess : stream:0x88d4840
:3:hip_stream.cpp           :268 : 159550578316 us: [pid:740193 tid: 0x778c75a53740] �[32m hipStreamCreateWithFlags ( 0x88d4b78, 1 ) �[0m
:3:rocdevice.cpp            :3036: 159550578323 us: [pid:740193 tid: 0x778c75a53740] Number of allocated hardware queues with low priority: 0, with normal priority: 1, with high priority: 0, maximum per priority is: 4
:3:rocdevice.cpp            :3114: 159550602074 us: [pid:740193 tid: 0x778c75a53740] Created SWq=0x778c75635000 to map on HWq=0x77843c200000 with size 16384 with priority 1, cooperative: 0
:3:rocdevice.cpp            :3207: 159550602103 us: [pid:740193 tid: 0x778c75a53740] acquireQueue refCount: 0x77843c200000 (1)
:4:rocdevice.cpp            :2371: 159550602112 us: [pid:740193 tid: 0x778c75a53740] Allocate hsa device memory 0x77843cb00000, size 0x100000
:4:rocdevice.cpp            :2210: 159550602610 us: [pid:740193 tid: 0x778c75a53740] Allocate hsa host memory 0x774c22000000, size 0x400000, numa_node = 0
:3:hip_stream.cpp           :274 : 159550602781 us: [pid:740193 tid: 0x778c75a53740] hipStreamCreateWithFlags: Returned hipSuccess : stream:0x88f8f20
:3:hip_device_runtime.cpp   :673 : 159550602791 us: [pid:740193 tid: 0x778c75a53740] �[32m hipGetDeviceCount ( 0x7ffd88c125e8 ) �[0m
:3:hip_device_runtime.cpp   :675 : 159550602794 us: [pid:740193 tid: 0x778c75a53740] hipGetDeviceCount: Returned hipSuccess : 
:3:hip_peer.cpp             :152 : 159550602806 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceCanAccessPeer ( 0x7ffd88c12598, 0, 1 ) �[0m
:3:hip_peer.cpp             :153 : 159550602811 us: [pid:740193 tid: 0x778c75a53740] hipDeviceCanAccessPeer: Returned hipSuccess : 
:3:hip_peer.cpp             :172 : 159550602814 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceEnablePeerAccess ( 1, 0 ) �[0m
:3:hip_peer.cpp             :186 : 159550602819 us: [pid:740193 tid: 0x778c75a53740] hipDeviceEnablePeerAccess: Returned hipSuccess : 
:3:hip_peer.cpp             :152 : 159550602823 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceCanAccessPeer ( 0x7ffd88c12598, 0, 2 ) �[0m
:3:hip_peer.cpp             :153 : 159550602826 us: [pid:740193 tid: 0x778c75a53740] hipDeviceCanAccessPeer: Returned hipSuccess : 
:3:hip_peer.cpp             :172 : 159550602829 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceEnablePeerAccess ( 2, 0 ) �[0m
:3:hip_peer.cpp             :186 : 159550602832 us: [pid:740193 tid: 0x778c75a53740] hipDeviceEnablePeerAccess: Returned hipSuccess : 
:3:hip_peer.cpp             :152 : 159550602837 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceCanAccessPeer ( 0x7ffd88c12598, 0, 3 ) �[0m
:3:hip_peer.cpp             :153 : 159550602840 us: [pid:740193 tid: 0x778c75a53740] hipDeviceCanAccessPeer: Returned hipSuccess : 
:3:hip_peer.cpp             :172 : 159550602843 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceEnablePeerAccess ( 3, 0 ) �[0m
:3:hip_peer.cpp             :186 : 159550602846 us: [pid:740193 tid: 0x778c75a53740] hipDeviceEnablePeerAccess: Returned hipSuccess : 
:3:hip_peer.cpp             :152 : 159550602849 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceCanAccessPeer ( 0x7ffd88c12598, 0, 4 ) �[0m
:3:hip_peer.cpp             :153 : 159550602852 us: [pid:740193 tid: 0x778c75a53740] hipDeviceCanAccessPeer: Returned hipSuccess : 
:3:hip_peer.cpp             :172 : 159550602855 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceEnablePeerAccess ( 4, 0 ) �[0m
:3:hip_peer.cpp             :186 : 159550602859 us: [pid:740193 tid: 0x778c75a53740] hipDeviceEnablePeerAccess: Returned hipSuccess : 
:3:hip_peer.cpp             :152 : 159550602862 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceCanAccessPeer ( 0x7ffd88c12598, 0, 5 ) �[0m
:3:hip_peer.cpp             :153 : 159550602866 us: [pid:740193 tid: 0x778c75a53740] hipDeviceCanAccessPeer: Returned hipSuccess : 
:3:hip_peer.cpp             :172 : 159550602869 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceEnablePeerAccess ( 5, 0 ) �[0m
:3:hip_peer.cpp             :186 : 159550602872 us: [pid:740193 tid: 0x778c75a53740] hipDeviceEnablePeerAccess: Returned hipSuccess : 
:3:hip_peer.cpp             :152 : 159550602875 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceCanAccessPeer ( 0x7ffd88c12598, 0, 6 ) �[0m
:3:hip_peer.cpp             :153 : 159550602879 us: [pid:740193 tid: 0x778c75a53740] hipDeviceCanAccessPeer: Returned hipSuccess : 
:3:hip_peer.cpp             :172 : 159550602882 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceEnablePeerAccess ( 6, 0 ) �[0m
:3:hip_peer.cpp             :186 : 159550602889 us: [pid:740193 tid: 0x778c75a53740] hipDeviceEnablePeerAccess: Returned hipSuccess : 
:3:hip_peer.cpp             :152 : 159550602892 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceCanAccessPeer ( 0x7ffd88c12598, 0, 7 ) �[0m
:3:hip_peer.cpp             :153 : 159550602896 us: [pid:740193 tid: 0x778c75a53740] hipDeviceCanAccessPeer: Returned hipSuccess : 
:3:hip_peer.cpp             :172 : 159550602899 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceEnablePeerAccess ( 7, 0 ) �[0m
:3:hip_peer.cpp             :186 : 159550602902 us: [pid:740193 tid: 0x778c75a53740] hipDeviceEnablePeerAccess: Returned hipSuccess : 
:3:hip_context.cpp          :279 : 159550602907 us: [pid:740193 tid: 0x778c75a53740] �[32m hipCtxPushCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :289 : 159550602910 us: [pid:740193 tid: 0x778c75a53740] hipCtxPushCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :262 : 159550602915 us: [pid:740193 tid: 0x778c75a53740] �[32m hipCtxPopCurrent ( char array:<null> ) �[0m
:3:hip_context.cpp          :275 : 159550602918 us: [pid:740193 tid: 0x778c75a53740] hipCtxPopCurrent: Returned hipSuccess : 
:3:hip_device_runtime.cpp   :161 : 159550602922 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetAttribute ( 0x7ffd88c125e8, 88, 0 ) �[0m
:3:hip_device_runtime.cpp   :452 : 159550602931 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetAttribute: Returned hipSuccess : 
:3:hip_mempool.cpp          :314 : 159550602945 us: [pid:740193 tid: 0x778c75a53740] �[32m hipMemPoolCreate ( 0x7ffd88c124d0, 0x7ffd88c124e8 ) �[0m
:3:hip_mempool.cpp          :338 : 159550602950 us: [pid:740193 tid: 0x778c75a53740] hipMemPoolCreate: Returned hipSuccess : 
:3:hip_mempool.cpp          :239 : 159550602954 us: [pid:740193 tid: 0x778c75a53740] �[32m hipMemPoolSetAttribute ( 0x892f4a0, 4, 0x7ffd88c124e0 ) �[0m
:3:hip_mempool.cpp          :244 : 159550602958 us: [pid:740193 tid: 0x778c75a53740] hipMemPoolSetAttribute: Returned hipSuccess : 
:3:hip_mempool.cpp          :314 : 159550602961 us: [pid:740193 tid: 0x778c75a53740] �[32m hipMemPoolCreate ( 0x7ffd88c124d0, 0x7ffd88c124e8 ) �[0m
:3:hip_mempool.cpp          :338 : 159550602964 us: [pid:740193 tid: 0x778c75a53740] hipMemPoolCreate: Returned hipSuccess : 
:3:hip_mempool.cpp          :239 : 159550602968 us: [pid:740193 tid: 0x778c75a53740] �[32m hipMemPoolSetAttribute ( 0x93d83c0, 4, 0x7ffd88c124e0 ) �[0m
:3:hip_mempool.cpp          :244 : 159550602970 us: [pid:740193 tid: 0x778c75a53740] hipMemPoolSetAttribute: Returned hipSuccess : 
:3:hip_device_runtime.cpp   :161 : 159550602974 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetAttribute ( 0x7ffd88c125d0, 9, 0 ) �[0m
:3:hip_device_runtime.cpp   :452 : 159550602977 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetAttribute: Returned hipSuccess : 
:3:hip_context.cpp          :279 : 159550603127 us: [pid:740193 tid: 0x778c75a53740] �[32m hipCtxPushCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :289 : 159550603136 us: [pid:740193 tid: 0x778c75a53740] hipCtxPushCurrent: Returned hipSuccess : 
:3:hip_memory.cpp           :762 : 159550603149 us: [pid:740193 tid: 0x778c75a53740] �[32m hipHostMalloc ( 0x7ffd88c123b8, 134217728, 6 ) �[0m
:4:rocdevice.cpp            :2210: 159550616933 us: [pid:740193 tid: 0x778c75a53740] Allocate hsa host memory 0x774bfda00000, size 0x8000000, numa_node = 0
:3:hip_memory.cpp           :768 : 159550617442 us: [pid:740193 tid: 0x778c75a53740] hipHostMalloc: Returned hipSuccess : 0x774bfda00000: duration: 14293 us
:3:hip_memory.cpp           :3828: 159550617465 us: [pid:740193 tid: 0x778c75a53740] �[32m hipHostGetDevicePointer ( 0x7ffd88c12368, 0x774bfda00000, 0 ) �[0m
:3:hip_memory.cpp           :3843: 159550617469 us: [pid:740193 tid: 0x778c75a53740] hipHostGetDevicePointer: Returned hipSuccess : 0x774bfda00000
:3:hip_context.cpp          :262 : 159550617476 us: [pid:740193 tid: 0x778c75a53740] �[32m hipCtxPopCurrent ( char array:<null> ) �[0m
:3:hip_context.cpp          :275 : 159550617479 us: [pid:740193 tid: 0x778c75a53740] hipCtxPopCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :191 : 159550617549 us: [pid:740193 tid: 0x778c75a53740] �[32m hipCtxSetCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :205 : 159550617556 us: [pid:740193 tid: 0x778c75a53740] hipCtxSetCurrent: Returned hipSuccess : 
:3:hip_event.cpp            :327 : 159550617560 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x9315210, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617566 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x93e8360
:3:hip_event.cpp            :327 : 159550617569 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x9315300, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617573 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x92a2b40
:3:hip_event.cpp            :327 : 159550617576 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x92bb530, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617579 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x92a2c00
:3:hip_event.cpp            :327 : 159550617582 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x92bb5b0, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617585 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x92bb5c0
:3:hip_event.cpp            :327 : 159550617589 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x92bb690, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617592 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x92e53e0
:3:hip_event.cpp            :327 : 159550617595 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x92e54b0, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617598 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x92e54c0
:3:hip_event.cpp            :327 : 159550617602 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x92e5590, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617606 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x92e55a0
:3:hip_event.cpp            :327 : 159550617609 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x92e5670, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617612 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x92e5680
:3:hip_event.cpp            :327 : 159550617615 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x92e5750, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617618 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x92e5760
:3:hip_event.cpp            :327 : 159550617621 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x92e5830, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617625 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x92e5840
:3:hip_event.cpp            :327 : 159550617628 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x88979e0, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617632 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x88979f0
:3:hip_event.cpp            :327 : 159550617635 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x8897ac0, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617638 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x8897ad0
:3:hip_event.cpp            :327 : 159550617641 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x8897ba0, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617644 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x8897bb0
:3:hip_event.cpp            :327 : 159550617647 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x8897ca0, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617655 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x8897cb0
:3:hip_event.cpp            :327 : 159550617659 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x8897da0, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617662 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x8897db0
:3:hip_event.cpp            :327 : 159550617665 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x9311890, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617668 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x93118a0
:3:hip_event.cpp            :327 : 159550617672 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x9311990, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617675 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x93119a0
:3:hip_event.cpp            :327 : 159550617678 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x9311a90, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617681 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x9311aa0
:3:hip_event.cpp            :327 : 159550617684 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x9311b90, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617688 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x9311ba0
:3:hip_event.cpp            :327 : 159550617691 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x9311c90, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617694 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x9311ca0
:3:hip_event.cpp            :327 : 159550617697 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x92be4f0, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617701 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x92be500
:3:hip_event.cpp            :327 : 159550617704 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x92be5f0, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617707 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x92be600
:3:hip_event.cpp            :327 : 159550617710 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x92be6f0, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617713 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x92be700
:3:hip_event.cpp            :327 : 159550617717 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x92be7f0, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617721 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x92be800
:3:hip_event.cpp            :327 : 159550617724 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x92be8f0, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617727 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x92be900
:3:hip_event.cpp            :327 : 159550617730 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x93dcad0, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617733 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x93dcae0
:3:hip_event.cpp            :327 : 159550617736 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x93dcbd0, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617739 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x93dcbe0
:3:hip_event.cpp            :327 : 159550617742 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x93dccd0, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617747 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x93dcce0
:3:hip_event.cpp            :327 : 159550617753 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x93dcdd0, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617756 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x93dcde0
:3:hip_event.cpp            :327 : 159550617760 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x93dced0, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617763 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x93dcee0
:3:hip_event.cpp            :327 : 159550617766 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x9268890, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617769 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x92688a0
:3:hip_event.cpp            :327 : 159550617772 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventCreateWithFlags ( 0x9268990, 2 ) �[0m
:3:hip_event.cpp            :333 : 159550617777 us: [pid:740193 tid: 0x778c75a53740] hipEventCreateWithFlags: Returned hipSuccess : event:0x92689a0
:3:hip_device_runtime.cpp   :161 : 159550617812 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetAttribute ( 0x7ffd88c0e6f0, 26, 0 ) �[0m
:3:hip_device_runtime.cpp   :452 : 159550617818 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetAttribute: Returned hipSuccess : 
:3:hip_device_runtime.cpp   :161 : 159550617821 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetAttribute ( 0x7ffd88c0e6f4, 27, 0 ) �[0m
:3:hip_device_runtime.cpp   :452 : 159550617824 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetAttribute: Returned hipSuccess : 
:3:hip_device_runtime.cpp   :161 : 159550617827 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetAttribute ( 0x7ffd88c0e6f8, 28, 0 ) �[0m
:3:hip_device_runtime.cpp   :452 : 159550617831 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetAttribute: Returned hipSuccess : 
:3:hip_device_runtime.cpp   :161 : 159550617835 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceGetAttribute ( 0x7ffd88c0e6fc, 74, 0 ) �[0m
:3:hip_device_runtime.cpp   :452 : 159550617838 us: [pid:740193 tid: 0x778c75a53740] hipDeviceGetAttribute: Returned hipSuccess : 
:3:hip_context.cpp          :279 : 159550617844 us: [pid:740193 tid: 0x778c75a53740] �[32m hipCtxPushCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :289 : 159550617847 us: [pid:740193 tid: 0x778c75a53740] hipCtxPushCurrent: Returned hipSuccess : 
:3:hip_module.cpp           :67  : 159550617859 us: [pid:740193 tid: 0x778c75a53740] �[32m hipModuleLoadDataEx ( 0x7ffd88c0e6e8, 0x7cdacd4 ) �[0m
:3:devprogram.cpp           :2658: 159550618041 us: [pid:740193 tid: 0x778c75a53740] Using Code Object V5.
:3:hip_module.cpp           :68  : 159550618535 us: [pid:740193 tid: 0x778c75a53740] hipModuleLoadDataEx: Returned hipSuccess : 
:3:hip_module.cpp           :76  : 159550618550 us: [pid:740193 tid: 0x778c75a53740] �[32m hipModuleGetFunction ( 0x7ffd88c0e710, 0x9225a90, conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g$async_dispatch_0_matmul_like_3748250x224x32_f32 ) �[0m
:3:hip_module.cpp           :90  : 159550618556 us: [pid:740193 tid: 0x778c75a53740] hipModuleGetFunction: Returned hipSuccess : 
:3:hip_module.cpp           :182 : 159550618566 us: [pid:740193 tid: 0x778c75a53740] �[32m hipFuncSetAttribute ( 0x92e3300, 8, 8960 ) �[0m
:3:hip_module.cpp           :228 : 159550618570 us: [pid:740193 tid: 0x778c75a53740] hipFuncSetAttribute: Returned hipSuccess : 
:3:hip_context.cpp          :262 : 159550618573 us: [pid:740193 tid: 0x778c75a53740] �[32m hipCtxPopCurrent ( char array:<null> ) �[0m
:3:hip_context.cpp          :275 : 159550618576 us: [pid:740193 tid: 0x778c75a53740] hipCtxPopCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :279 : 159551187971 us: [pid:740193 tid: 0x778c75a53740] �[32m hipCtxPushCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :289 : 159551188020 us: [pid:740193 tid: 0x778c75a53740] hipCtxPushCurrent: Returned hipSuccess : 
:3:hip_memory.cpp           :756 : 159551188037 us: [pid:740193 tid: 0x778c75a53740] �[32m hipMalloc ( 0x7ffd88c12848, 479776000 ) �[0m
:4:rocdevice.cpp            :2371: 159551188169 us: [pid:740193 tid: 0x778c75a53740] Allocate hsa device memory 0x774bc4600000, size 0x1c98cd00
:3:rocdevice.cpp            :2410: 159551188175 us: [pid:740193 tid: 0x778c75a53740] Device=0x883a030, freeMem_ = 0x2fe2673300
:3:hip_memory.cpp           :758 : 159551188181 us: [pid:740193 tid: 0x778c75a53740] hipMalloc: Returned hipSuccess : 0x774bc4600000: duration: 144 us
:3:hip_context.cpp          :262 : 159551188188 us: [pid:740193 tid: 0x778c75a53740] �[32m hipCtxPopCurrent ( char array:<null> ) �[0m
:3:hip_context.cpp          :275 : 159551188192 us: [pid:740193 tid: 0x778c75a53740] hipCtxPopCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :279 : 159551188196 us: [pid:740193 tid: 0x778c75a53740] �[32m hipCtxPushCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :289 : 159551188199 us: [pid:740193 tid: 0x778c75a53740] hipCtxPushCurrent: Returned hipSuccess : 
:3:hip_memory.cpp           :762 : 159551188204 us: [pid:740193 tid: 0x778c75a53740] �[32m hipHostMalloc ( 0x7ffd88c127a8, 479776000, 6 ) �[0m
:4:rocdevice.cpp            :2210: 159551237845 us: [pid:740193 tid: 0x778c75a53740] Allocate hsa host memory 0x774ba7c00000, size 0x1c98cd00, numa_node = 0
:3:hip_memory.cpp           :768 : 159551239259 us: [pid:740193 tid: 0x778c75a53740] hipHostMalloc: Returned hipSuccess : 0x774ba7c00000: duration: 51055 us
:3:hip_memory.cpp           :3828: 159551239273 us: [pid:740193 tid: 0x778c75a53740] �[32m hipHostGetDevicePointer ( 0x7ffd88c12758, 0x774ba7c00000, 0 ) �[0m
:3:hip_memory.cpp           :3843: 159551239277 us: [pid:740193 tid: 0x778c75a53740] hipHostGetDevicePointer: Returned hipSuccess : 0x774ba7c00000
:3:hip_context.cpp          :262 : 159551239284 us: [pid:740193 tid: 0x778c75a53740] �[32m hipCtxPopCurrent ( char array:<null> ) �[0m
:3:hip_context.cpp          :275 : 159551239287 us: [pid:740193 tid: 0x778c75a53740] hipCtxPopCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :279 : 159551264873 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxPushCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :289 : 159551264938 us: [pid:740193 tid: 0x774c20ffd640] hipCtxPushCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :279 : 159551264950 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxPushCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :289 : 159551264958 us: [pid:740193 tid: 0x774c20ffd640] hipCtxPushCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :262 : 159551264974 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxPopCurrent ( char array:<null> ) �[0m
:3:hip_context.cpp          :275 : 159551264982 us: [pid:740193 tid: 0x774c20ffd640] hipCtxPopCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :191 : 159551264992 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxSetCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :205 : 159551264999 us: [pid:740193 tid: 0x774c20ffd640] hipCtxSetCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :191 : 159551265043 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxSetCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :205 : 159551265050 us: [pid:740193 tid: 0x774c20ffd640] hipCtxSetCurrent: Returned hipSuccess : 
:3:hip_memory.cpp           :1631: 159551265081 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipMemcpyAsync ( 0x774bc4600000, 0x774ba7c00000, 479776000, hipMemcpyDeviceToDevice, stream:0x88d4840 ) �[0m
:4:command.cpp              :358 : 159551265106 us: [pid:740193 tid: 0x774c20ffd640] Command (CopyDeviceToDevice) enqueued: 0x774ba0009870
:4:rocblit.cpp              :503 : 159551265135 us: [pid:740193 tid: 0x774c20ffd640] Last copy mask 0x0
:4:rocblit.cpp              :509 : 159551265148 us: [pid:740193 tid: 0x774c20ffd640] Query copy engine status 0, free_engine mask 0xffff
:4:rocblit.cpp              :520 : 159551265173 us: [pid:740193 tid: 0x774c20ffd640] HSA Async Copy on copy_engine=0x1, dst=0x774bc4600000, src=0x774ba7c00000, size=479776000, forceSDMA=0, wait_event=0x0, completion_signal=0x778c3fbff380
:3:hip_memory.cpp           :1632: 159551266467 us: [pid:740193 tid: 0x774c20ffd640] hipMemcpyAsync: Returned hipSuccess : : duration: 1386 us
:3:hip_context.cpp          :191 : 159551266480 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxSetCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :205 : 159551266484 us: [pid:740193 tid: 0x774c20ffd640] hipCtxSetCurrent: Returned hipSuccess : 
:3:hip_event.cpp            :423 : 159551266490 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipEventRecord ( event:0x92689a0, stream:0x88d4840 ) �[0m
:4:command.cpp              :358 : 159551266495 us: [pid:740193 tid: 0x774c20ffd640] Command (InternalMarker) enqueued: 0x774ba0109fb0
:4:rocvirtual.cpp           :1212: 159551266510 us: [pid:740193 tid: 0x774c20ffd640] SWq=0x778c75716000, HWq=0x77843ce00000, id=1, BarrierValue Header = 0x1500 AmdFormat = 0x2 (type=0, barrier=1, acquire=2, release=2), signal=0x778c3fbff380, value = 0x1 mask = 0x7fffffffffffffff cond: LT, completion_signal=0x778c3fbff300
:3:hip_event.cpp            :424 : 159551266513 us: [pid:740193 tid: 0x774c20ffd640] hipEventRecord: Returned hipSuccess : 
:3:hip_event.cpp            :423 : 159551266517 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipEventRecord ( event:0x92688a0, stream:0x88d4840 ) �[0m
:4:command.cpp              :358 : 159551266521 us: [pid:740193 tid: 0x774c20ffd640] Command (InternalMarker) enqueued: 0x774ba010a240
:4:rocvirtual.cpp           :1212: 159551266525 us: [pid:740193 tid: 0x774c20ffd640] SWq=0x778c75716000, HWq=0x77843ce00000, id=1, BarrierValue Header = 0x1500 AmdFormat = 0x2 (type=0, barrier=1, acquire=2, release=2), signal=0x0, value = 0x0 mask = 0x0 cond: EQ, completion_signal=0x778c3fbff280
:3:hip_event.cpp            :424 : 159551266528 us: [pid:740193 tid: 0x774c20ffd640] hipEventRecord: Returned hipSuccess : 
:3:hip_context.cpp          :262 : 159551266537 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxPopCurrent ( char array:<null> ) �[0m
:3:hip_context.cpp          :275 : 159551266541 us: [pid:740193 tid: 0x774c20ffd640] hipCtxPopCurrent: Returned hipSuccess : 
:3:hip_event.cpp            :434 : 159551266773 us: [pid:740193 tid: 0x774c21fff640] �[32m hipEventSynchronize ( event:0x92688a0 ) �[0m
:3:rocvirtual.hpp           :56  : 159551266827 us: [pid:740193 tid: 0x774c21fff640] Host active wait for Signal = (0x778c3fbff280) for -1 ns
:3:hip_event.cpp            :454 : 159551275932 us: [pid:740193 tid: 0x774c21fff640] hipEventSynchronize: Returned hipSuccess : 
:3:hip_event.cpp            :474 : 159551275956 us: [pid:740193 tid: 0x774c21fff640] �[32m hipEventQuery ( event:0x92689a0 ) �[0m
:3:hip_event.cpp            :475 : 159551275961 us: [pid:740193 tid: 0x774c21fff640] hipEventQuery: Returned hipSuccess : 
:3:hip_memory.cpp           :836 : 159551276017 us: [pid:740193 tid: 0x774c21fff640] �[32m hipHostFree ( 0x774ba7c00000 ) �[0m
:4:commandqueue.cpp         :160 : 159551276027 us: [pid:740193 tid: 0x774c21fff640] Marker queued to ensure finish
:4:command.cpp              :358 : 159551276031 us: [pid:740193 tid: 0x774c21fff640] Command (InternalMarker) enqueued: 0x774b980015e0
:3:rocvirtual.cpp           :528 : 159551276056 us: [pid:740193 tid: 0x774c21fff640] Set Handler: handle(0x778c3fbfd200), timestamp(0x8901a10)
:4:rocvirtual.cpp           :1133: 159551276062 us: [pid:740193 tid: 0x774c21fff640] SWq=0x778c75716000, HWq=0x77843ce00000, id=1, BarrierAND Header = 0x1503 (type=3, barrier=1, acquire=2, release=2), dep_signal=[0x0, 0x0, 0x0, 0x0, 0x0], completion_signal=0x778c3fbfd200
:4:commandqueue.cpp         :166 : 159551276065 us: [pid:740193 tid: 0x774c21fff640] HW Event not ready, awaiting completion instead
:4:command.cpp              :243 : 159551276069 us: [pid:740193 tid: 0x774c21fff640] Waiting for event 0x774b980015e0 to complete, current status 2
:3:rocvirtual.cpp           :226 : 159551276212 us: [pid:740193 tid: 0x77843d7ff640] Handler: value(0), timestamp(0x774ba00096b0), handle(0x778c3fbfd200)
:4:command.cpp              :169 : 159551276261 us: [pid:740193 tid: 0x77843d7ff640] Command 0x774ba0009870 complete
:3:rocvirtual.cpp           :165 : 159551276301 us: [pid:740193 tid: 0x77843d7ff640] Signal = (0x778c3fbff300), Translated start/end = 159551266523964 / 159551275923161, Elapsed = 9399197 ns, ticks start/end = 15942518811333 / 15942519749841, Ticks elapsed = 938508
:4:command.cpp              :163 : 159551276311 us: [pid:740193 tid: 0x77843d7ff640] Command 0x774ba0109fb0 complete (Wall: 9425, CPU: 0, GPU: 9399 us)
:3:rocvirtual.cpp           :165 : 159551276320 us: [pid:740193 tid: 0x77843d7ff640] Signal = (0x778c3fbff280), Translated start/end = 159551275926166 / 159551275927608, Elapsed = 1442 ns, ticks start/end = 15942519750141 / 15942519750285, Ticks elapsed = 144
:4:command.cpp              :163 : 159551276328 us: [pid:740193 tid: 0x77843d7ff640] Command 0x774ba010a240 complete (Wall: 9429, CPU: 0, GPU: 1 us)
:4:command.cpp              :258 : 159551276336 us: [pid:740193 tid: 0x774c21fff640] Event 0x774b980015e0 wait completed
:4:commandqueue.cpp         :183 : 159551276344 us: [pid:740193 tid: 0x774c21fff640] All commands finished
:4:command.cpp              :163 : 159551276336 us: [pid:740193 tid: 0x77843d7ff640] Command 0x774b980015e0 complete (Wall: 9836, CPU: 0, GPU: 298 us)
:3:hip_context.cpp          :279 : 159551311878 us: [pid:740193 tid: 0x778c75a53740] �[32m hipCtxPushCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :289 : 159551311911 us: [pid:740193 tid: 0x778c75a53740] hipCtxPushCurrent: Returned hipSuccess : 
:3:hip_memory.cpp           :756 : 159551311917 us: [pid:740193 tid: 0x778c75a53740] �[32m hipMalloc ( 0x7ffd88c12848, 28672 ) �[0m
:4:rocdevice.cpp            :2387: 159551315423 us: [pid:740193 tid: 0x774c21fff640] Free hsa memory 0x774ba7c00000
:3:hip_memory.cpp           :845 : 159551315469 us: [pid:740193 tid: 0x774c21fff640] hipHostFree: Returned hipSuccess : 
:4:rocdevice.cpp            :2371: 159551315503 us: [pid:740193 tid: 0x778c75a53740] Allocate hsa device memory 0x774c20200000, size 0x7000
:3:rocdevice.cpp            :2410: 159551315511 us: [pid:740193 tid: 0x778c75a53740] Device=0x883a030, freeMem_ = 0x2fe266c300
:3:hip_memory.cpp           :758 : 159551315518 us: [pid:740193 tid: 0x778c75a53740] hipMalloc: Returned hipSuccess : 0x774c20200000: duration: 3601 us
:3:hip_context.cpp          :262 : 159551315523 us: [pid:740193 tid: 0x778c75a53740] �[32m hipCtxPopCurrent ( char array:<null> ) �[0m
:3:hip_context.cpp          :275 : 159551315527 us: [pid:740193 tid: 0x778c75a53740] hipCtxPopCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :279 : 159551315594 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxPushCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :289 : 159551315619 us: [pid:740193 tid: 0x774c20ffd640] hipCtxPushCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :279 : 159551315625 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxPushCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :289 : 159551315628 us: [pid:740193 tid: 0x774c20ffd640] hipCtxPushCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :262 : 159551315634 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxPopCurrent ( char array:<null> ) �[0m
:3:hip_context.cpp          :275 : 159551315637 us: [pid:740193 tid: 0x774c20ffd640] hipCtxPopCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :191 : 159551315642 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxSetCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :205 : 159551315645 us: [pid:740193 tid: 0x774c20ffd640] hipCtxSetCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :191 : 159551315650 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxSetCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :205 : 159551315653 us: [pid:740193 tid: 0x774c20ffd640] hipCtxSetCurrent: Returned hipSuccess : 
:3:hip_memory.cpp           :1644: 159551315716 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipMemcpyHtoDAsync ( 0x774c20200000, 0x774ba010a620, 28672, stream:0x88d4840 ) �[0m
:4:command.cpp              :358 : 159551315735 us: [pid:740193 tid: 0x774c20ffd640] Command (CopyHostToDevice) enqueued: 0x774ba0112630
:4:rocblit.cpp              :1803: 159551315743 us: [pid:740193 tid: 0x774c20ffd640] Unpinned write path, Async = 1
:4:rocblit.cpp              :633 : 159551315747 us: [pid:740193 tid: 0x774c20ffd640] HSA Copy Using Staging resource size 28672
:4:rocblit.cpp              :680 : 159551315754 us: [pid:740193 tid: 0x774c20ffd640] HSA Async Copy staged H2D
:4:rocblit.cpp              :503 : 159551315758 us: [pid:740193 tid: 0x774c20ffd640] Last copy mask 0x0
:4:rocblit.cpp              :509 : 159551315764 us: [pid:740193 tid: 0x774c20ffd640] Query copy engine status 0, free_engine mask 0xffff
:4:rocblit.cpp              :520 : 159551315767 us: [pid:740193 tid: 0x774c20ffd640] HSA Async Copy on copy_engine=0x1, dst=0x774c20200000, src=0x77843c400000, size=28672, forceSDMA=0, wait_event=0x0, completion_signal=0x778c3fbff180
:3:hip_memory.cpp           :1657: 159551315774 us: [pid:740193 tid: 0x774c20ffd640] hipMemcpyHtoDAsync: Returned hipSuccess : : duration: 58 us
:3:hip_context.cpp          :191 : 159551315778 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxSetCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :205 : 159551315783 us: [pid:740193 tid: 0x774c20ffd640] hipCtxSetCurrent: Returned hipSuccess : 
:3:hip_event.cpp            :423 : 159551315789 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipEventRecord ( event:0x92689a0, stream:0x88d4840 ) �[0m
:4:command.cpp              :358 : 159551315793 us: [pid:740193 tid: 0x774c20ffd640] Command (InternalMarker) enqueued: 0x774ba0112890
:4:rocvirtual.cpp           :1212: 159551315800 us: [pid:740193 tid: 0x774c20ffd640] SWq=0x778c75716000, HWq=0x77843ce00000, id=1, BarrierValue Header = 0x1500 AmdFormat = 0x2 (type=0, barrier=1, acquire=2, release=2), signal=0x0, value = 0x0 mask = 0x0 cond: EQ, completion_signal=0x778c3fbff100
:3:hip_event.cpp            :424 : 159551315805 us: [pid:740193 tid: 0x774c20ffd640] hipEventRecord: Returned hipSuccess : 
:3:hip_event.cpp            :423 : 159551315808 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipEventRecord ( event:0x92688a0, stream:0x88d4840 ) �[0m
:4:command.cpp              :358 : 159551315812 us: [pid:740193 tid: 0x774c20ffd640] Command (InternalMarker) enqueued: 0x774ba0109fb0
:4:rocvirtual.cpp           :1212: 159551315816 us: [pid:740193 tid: 0x774c20ffd640] SWq=0x778c75716000, HWq=0x77843ce00000, id=1, BarrierValue Header = 0x1500 AmdFormat = 0x2 (type=0, barrier=1, acquire=2, release=2), signal=0x0, value = 0x0 mask = 0x0 cond: EQ, completion_signal=0x778c3fbff080
:3:hip_event.cpp            :424 : 159551315819 us: [pid:740193 tid: 0x774c20ffd640] hipEventRecord: Returned hipSuccess : 
:3:hip_context.cpp          :262 : 159551315826 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxPopCurrent ( char array:<null> ) �[0m
:3:hip_context.cpp          :275 : 159551315828 us: [pid:740193 tid: 0x774c20ffd640] hipCtxPopCurrent: Returned hipSuccess : 
:3:hip_event.cpp            :434 : 159551315839 us: [pid:740193 tid: 0x774c21fff640] �[32m hipEventSynchronize ( event:0x92688a0 ) �[0m
:3:hip_event.cpp            :454 : 159551315851 us: [pid:740193 tid: 0x774c21fff640] hipEventSynchronize: Returned hipSuccess : 
:3:hip_event.cpp            :474 : 159551315860 us: [pid:740193 tid: 0x774c21fff640] �[32m hipEventQuery ( event:0x92689a0 ) �[0m
:3:hip_event.cpp            :475 : 159551315868 us: [pid:740193 tid: 0x774c21fff640] hipEventQuery: Returned hipSuccess : 
EXEC @conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g
:3:hip_context.cpp          :279 : 159551315928 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxPushCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :289 : 159551315933 us: [pid:740193 tid: 0x774c20ffd640] hipCtxPushCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :279 : 159551315941 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxPushCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :289 : 159551315944 us: [pid:740193 tid: 0x774c20ffd640] hipCtxPushCurrent: Returned hipSuccess : 
:3:hip_memory.cpp           :756 : 159551315949 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipMalloc ( 0x774c20ffcd80, 3358432000 ) �[0m
:4:rocdevice.cpp            :2371: 159551316319 us: [pid:740193 tid: 0x774c20ffd640] Allocate hsa device memory 0x774acfa00000, size 0xc82d9b00
:3:rocdevice.cpp            :2410: 159551316339 us: [pid:740193 tid: 0x774c20ffd640] Device=0x883a030, freeMem_ = 0x2f1a392800
:3:hip_memory.cpp           :758 : 159551316348 us: [pid:740193 tid: 0x774c20ffd640] hipMalloc: Returned hipSuccess : 0x774acfa00000: duration: 399 us
:3:hip_context.cpp          :262 : 159551316354 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxPopCurrent ( char array:<null> ) �[0m
:3:hip_context.cpp          :275 : 159551316359 us: [pid:740193 tid: 0x774c20ffd640] hipCtxPopCurrent: Returned hipSuccess : 
:3:hip_event.cpp            :423 : 159551316366 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipEventRecord ( event:0x92689a0, stream:0x88f8f20 ) �[0m
:4:command.cpp              :358 : 159551316373 us: [pid:740193 tid: 0x774c20ffd640] Command (InternalMarker) enqueued: 0x774ba010a240
:4:rocvirtual.cpp           :1212: 159551316384 us: [pid:740193 tid: 0x774c20ffd640] SWq=0x778c75635000, HWq=0x77843c200000, id=2, BarrierValue Header = 0x1500 AmdFormat = 0x2 (type=0, barrier=1, acquire=2, release=2), signal=0x0, value = 0x0 mask = 0x0 cond: EQ, completion_signal=0x778c3fbfcb80
:3:hip_event.cpp            :424 : 159551316390 us: [pid:740193 tid: 0x774c20ffd640] hipEventRecord: Returned hipSuccess : 
:3:hip_event.cpp            :423 : 159551316397 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipEventRecord ( event:0x92688a0, stream:0x88d4840 ) �[0m
:4:command.cpp              :358 : 159551316402 us: [pid:740193 tid: 0x774c20ffd640] Command (InternalMarker) enqueued: 0x774ba0113330
:4:rocvirtual.cpp           :1212: 159551316410 us: [pid:740193 tid: 0x774c20ffd640] SWq=0x778c75716000, HWq=0x77843ce00000, id=1, BarrierValue Header = 0x1500 AmdFormat = 0x2 (type=0, barrier=1, acquire=2, release=2), signal=0x0, value = 0x0 mask = 0x0 cond: EQ, completion_signal=0x778c3fbff000
:3:hip_event.cpp            :424 : 159551316415 us: [pid:740193 tid: 0x774c20ffd640] hipEventRecord: Returned hipSuccess : 
:3:hip_context.cpp          :262 : 159551316424 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxPopCurrent ( char array:<null> ) �[0m
:3:hip_context.cpp          :275 : 159551316431 us: [pid:740193 tid: 0x774c20ffd640] hipCtxPopCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :279 : 159551316439 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxPushCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :289 : 159551316448 us: [pid:740193 tid: 0x774c20ffd640] hipCtxPushCurrent: Returned hipSuccess : 
:3:hip_event.cpp            :434 : 159551316431 us: [pid:740193 tid: 0x774c21fff640] �[32m hipEventSynchronize ( event:0x92688a0 ) �[0m
:3:hip_event.cpp            :454 : 159551316472 us: [pid:740193 tid: 0x774c21fff640] hipEventSynchronize: Returned hipSuccess : 
:3:hip_stream.cpp           :501 : 159551316459 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipStreamWaitEvent ( stream:0x88d4840, event:0x92689a0, 0 ) �[0m
:3:hip_stream.cpp           :502 : 159551316496 us: [pid:740193 tid: 0x774c20ffd640] hipStreamWaitEvent: Returned hipSuccess : 
:3:hip_context.cpp          :279 : 159551316505 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxPushCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :289 : 159551316510 us: [pid:740193 tid: 0x774c20ffd640] hipCtxPushCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :262 : 159551316519 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxPopCurrent ( char array:<null> ) �[0m
:3:hip_context.cpp          :275 : 159551316527 us: [pid:740193 tid: 0x774c20ffd640] hipCtxPopCurrent: Returned hipSuccess : 
:3:hip_event.cpp            :474 : 159551316511 us: [pid:740193 tid: 0x774c21fff640] �[32m hipEventQuery ( event:0x92689a0 ) �[0m
:3:hip_event.cpp            :475 : 159551316552 us: [pid:740193 tid: 0x774c21fff640] hipEventQuery: Returned hipSuccess : 
:3:hip_context.cpp          :191 : 159551316534 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxSetCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :205 : 159551316570 us: [pid:740193 tid: 0x774c20ffd640] hipCtxSetCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :191 : 159551316579 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxSetCurrent ( context:0x88cfb10 ) �[0m
:3:hip_context.cpp          :205 : 159551316584 us: [pid:740193 tid: 0x774c20ffd640] hipCtxSetCurrent: Returned hipSuccess : 
:3:hip_module.cpp           :500 : 159551316598 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipModuleLaunchKernel ( 0x0x92e3300, 7, 117133, 1, 256, 1, 1, 8960, stream:0x88d4840, 0x8ba8290, char array:<null> ) �[0m
:3:hip_module.cpp           :513 : 159551316605 us: [pid:740193 tid: 0x774c20ffd640] hipModuleLaunchKernel: Returned hipErrorInvalidValue : 
:3:hip_context.cpp          :262 : 159551316628 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipCtxPopCurrent ( char array:<null> ) �[0m
:3:hip_context.cpp          :275 : 159551316634 us: [pid:740193 tid: 0x774c20ffd640] hipCtxPopCurrent: Returned hipSuccess : 
:3:hip_module.cpp           :46  : 159551316652 us: [pid:740193 tid: 0x778c75a53740] �[32m hipModuleUnload ( 0x9225a90 ) �[0m
:3:hip_memory.cpp           :772 : 159551316653 us: [pid:740193 tid: 0x774c21fff640] �[32m hipFree ( 0x774bc4600000 ) �[0m
:4:commandqueue.cpp         :160 : 159551316677 us: [pid:740193 tid: 0x774c21fff640] Marker queued to ensure finish
:4:command.cpp              :358 : 159551316683 us: [pid:740193 tid: 0x774c21fff640] Command (InternalMarker) enqueued: 0x774b98001800
:3:rocvirtual.cpp           :528 : 159551316704 us: [pid:740193 tid: 0x774c21fff640] Set Handler: handle(0x778c3fbfaa00), timestamp(0x93ef150)
:4:rocvirtual.cpp           :1133: 159551316714 us: [pid:740193 tid: 0x774c21fff640] SWq=0x778c75635000, HWq=0x77843c200000, id=2, BarrierAND Header = 0x1503 (type=3, barrier=1, acquire=2, release=2), dep_signal=[0x0, 0x0, 0x0, 0x0, 0x0], completion_signal=0x778c3fbfaa00
:4:commandqueue.cpp         :166 : 159551316720 us: [pid:740193 tid: 0x774c21fff640] HW Event not ready, awaiting completion instead
:4:command.cpp              :243 : 159551316728 us: [pid:740193 tid: 0x774c21fff640] Waiting for event 0x774b98001800 to complete, current status 2
:3:rocvirtual.cpp           :226 : 159551316783 us: [pid:740193 tid: 0x77843d7ff640] Handler: value(0), timestamp(0x774ba010a5b0), handle(0x778c3fbfaa00)
:3:rocvirtual.cpp           :165 : 159551316812 us: [pid:740193 tid: 0x77843d7ff640] Signal = (0x778c3fbfcb80), Translated start/end = 159551316391414 / 159551316395981, Elapsed = 4567 ns, ticks start/end = 15942523790588 / 15942523791044, Ticks elapsed = 456
:4:command.cpp              :163 : 159551316823 us: [pid:740193 tid: 0x77843d7ff640] Command 0x774ba010a240 complete (Wall: 49898, CPU: 0, GPU: 4 us)
:4:command.cpp              :163 : 159551316832 us: [pid:740193 tid: 0x77843d7ff640] Command 0x774b98001800 complete (Wall: 50332, CPU: 0, GPU: 140 us)
:4:command.cpp              :258 : 159551316832 us: [pid:740193 tid: 0x774c21fff640] Event 0x774b98001800 wait completed
:4:commandqueue.cpp         :183 : 159551316856 us: [pid:740193 tid: 0x774c21fff640] All commands finished
:4:commandqueue.cpp         :160 : 159551316862 us: [pid:740193 tid: 0x774c21fff640] Marker queued to ensure finish
:4:command.cpp              :358 : 159551316868 us: [pid:740193 tid: 0x774c21fff640] Command (InternalMarker) enqueued: 0x774b98001800
:3:rocvirtual.cpp           :528 : 159551316880 us: [pid:740193 tid: 0x774c21fff640] Set Handler: handle(0x778c3fbfd280), timestamp(0x92c3090)
:4:rocvirtual.cpp           :1133: 159551316887 us: [pid:740193 tid: 0x774c21fff640] SWq=0x778c75716000, HWq=0x77843ce00000, id=1, BarrierAND Header = 0x1503 (type=3, barrier=1, acquire=2, release=2), dep_signal=[0x0, 0x0, 0x0, 0x0, 0x0], completion_signal=0x778c3fbfd280
:4:commandqueue.cpp         :166 : 159551316899 us: [pid:740193 tid: 0x774c21fff640] HW Event not ready, awaiting completion instead
:4:command.cpp              :243 : 159551316905 us: [pid:740193 tid: 0x774c21fff640] Waiting for event 0x774b98001800 to complete, current status 2
:3:rocvirtual.cpp           :226 : 159551316960 us: [pid:740193 tid: 0x77843d7ff640] Handler: value(0), timestamp(0x92c0fc0), handle(0x778c3fbfd280)
:4:command.cpp              :169 : 159551316981 us: [pid:740193 tid: 0x77843d7ff640] Command 0x774ba0112630 complete
:3:rocvirtual.cpp           :165 : 159551316990 us: [pid:740193 tid: 0x77843d7ff640] Signal = (0x778c3fbff100), Translated start/end = 159551315809290 / 159551315812655, Elapsed = 3365 ns, ticks start/end = 15942523732463 / 15942523732799, Ticks elapsed = 336
:4:command.cpp              :163 : 159551316997 us: [pid:740193 tid: 0x77843d7ff640] Command 0x774ba0112890 complete (Wall: 49314, CPU: 0, GPU: 3 us)
:3:rocvirtual.cpp           :165 : 159551317018 us: [pid:740193 tid: 0x77843d7ff640] Signal = (0x778c3fbff080), Translated start/end = 159551315823751 / 159551315828879, Elapsed = 5128 ns, ticks start/end = 15942523733907 / 15942523734419, Ticks elapsed = 512
:4:command.cpp              :163 : 159551317038 us: [pid:740193 tid: 0x77843d7ff640] Command 0x774ba0109fb0 complete (Wall: 49330, CPU: 0, GPU: 5 us)
:3:rocvirtual.cpp           :165 : 159551317046 us: [pid:740193 tid: 0x77843d7ff640] Signal = (0x778c3fbff000), Translated start/end = 159551316417313 / 159551316422080, Elapsed = 4767 ns, ticks start/end = 15942523793174 / 15942523793650, Ticks elapsed = 476
:4:command.cpp              :163 : 159551317052 us: [pid:740193 tid: 0x77843d7ff640] Command 0x774ba0113330 complete (Wall: 49924, CPU: 0, GPU: 4 us)
:4:command.cpp              :163 : 159551317058 us: [pid:740193 tid: 0x77843d7ff640] Command 0x774b98001800 complete (Wall: 50559, CPU: 0, GPU: 183 us)
:4:command.cpp              :258 : 159551317060 us: [pid:740193 tid: 0x774c21fff640] Event 0x774b98001800 wait completed
:4:commandqueue.cpp         :183 : 159551317079 us: [pid:740193 tid: 0x774c21fff640] All commands finished
:3:hip_module.cpp           :50  : 159551317832 us: [pid:740193 tid: 0x778c75a53740] hipModuleUnload: Returned hipSuccess : 
:4:rocdevice.cpp            :2387: 159551318089 us: [pid:740193 tid: 0x774c21fff640] Free hsa memory 0x774bc4600000
:3:rocdevice.cpp            :2410: 159551318105 us: [pid:740193 tid: 0x774c21fff640] Device=0x883a030, freeMem_ = 0x2f36d1f500
:3:hip_memory.cpp           :774 : 159551318113 us: [pid:740193 tid: 0x774c21fff640] hipFree: Returned hipSuccess : 
:3:hip_memory.cpp           :772 : 159551318121 us: [pid:740193 tid: 0x774c21fff640] �[32m hipFree ( 0x774c20200000 ) �[0m
:4:rocdevice.cpp            :2387: 159551318134 us: [pid:740193 tid: 0x774c21fff640] Free hsa memory 0x774c20200000
:3:rocdevice.cpp            :2410: 159551318140 us: [pid:740193 tid: 0x774c21fff640] Device=0x883a030, freeMem_ = 0x2f36d26500
:3:hip_memory.cpp           :774 : 159551318147 us: [pid:740193 tid: 0x774c21fff640] hipFree: Returned hipSuccess : 
:3:hip_memory.cpp           :836 : 159551318278 us: [pid:740193 tid: 0x778c75a53740] �[32m hipHostFree ( 0x774bfda00000 ) �[0m
:4:rocdevice.cpp            :2387: 159551319791 us: [pid:740193 tid: 0x778c75a53740] Free hsa memory 0x774bfda00000
:3:hip_memory.cpp           :845 : 159551319807 us: [pid:740193 tid: 0x778c75a53740] hipHostFree: Returned hipSuccess : 
:3:hip_memory.cpp           :772 : 159551319812 us: [pid:740193 tid: 0x778c75a53740] �[32m hipFree ( 0x774acfa00000 ) �[0m
:4:rocdevice.cpp            :2387: 159551320111 us: [pid:740193 tid: 0x778c75a53740] Free hsa memory 0x774acfa00000
:3:rocdevice.cpp            :2410: 159551320120 us: [pid:740193 tid: 0x778c75a53740] Device=0x883a030, freeMem_ = 0x2fff000000
:3:hip_memory.cpp           :774 : 159551320124 us: [pid:740193 tid: 0x778c75a53740] hipFree: Returned hipSuccess : 
:3:hip_mempool.cpp          :343 : 159551320129 us: [pid:740193 tid: 0x778c75a53740] �[32m hipMemPoolDestroy ( 0x892f4a0 ) �[0m
:3:hip_mempool.cpp          :366 : 159551320143 us: [pid:740193 tid: 0x778c75a53740] hipMemPoolDestroy: Returned hipSuccess : 
:3:hip_mempool.cpp          :343 : 159551320147 us: [pid:740193 tid: 0x778c75a53740] �[32m hipMemPoolDestroy ( 0x93d83c0 ) �[0m
:3:hip_mempool.cpp          :366 : 159551320150 us: [pid:740193 tid: 0x778c75a53740] hipMemPoolDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320154 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x93e8360 ) �[0m
:3:hip_event.cpp            :367 : 159551320159 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320163 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x92a2b40 ) �[0m
:3:hip_event.cpp            :367 : 159551320166 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320170 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x92a2c00 ) �[0m
:3:hip_event.cpp            :367 : 159551320173 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320176 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x92bb5c0 ) �[0m
:3:hip_event.cpp            :367 : 159551320179 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320182 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x92e53e0 ) �[0m
:3:hip_event.cpp            :367 : 159551320185 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320188 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x92e54c0 ) �[0m
:3:hip_event.cpp            :367 : 159551320192 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320196 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x92e55a0 ) �[0m
:3:hip_event.cpp            :367 : 159551320199 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320202 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x92e5680 ) �[0m
:3:hip_event.cpp            :367 : 159551320205 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320208 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x92e5760 ) �[0m
:3:hip_event.cpp            :367 : 159551320211 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320214 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x92e5840 ) �[0m
:3:hip_event.cpp            :367 : 159551320217 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320221 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x88979f0 ) �[0m
:3:hip_event.cpp            :367 : 159551320224 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320227 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x8897ad0 ) �[0m
:3:hip_event.cpp            :367 : 159551320230 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320233 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x8897bb0 ) �[0m
:3:hip_event.cpp            :367 : 159551320236 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320239 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x8897cb0 ) �[0m
:3:hip_event.cpp            :367 : 159551320242 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320245 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x8897db0 ) �[0m
:3:hip_event.cpp            :367 : 159551320253 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320256 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x93118a0 ) �[0m
:3:hip_event.cpp            :367 : 159551320259 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320262 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x93119a0 ) �[0m
:3:hip_event.cpp            :367 : 159551320265 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320268 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x9311aa0 ) �[0m
:3:hip_event.cpp            :367 : 159551320270 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320273 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x9311ba0 ) �[0m
:3:hip_event.cpp            :367 : 159551320276 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320281 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x9311ca0 ) �[0m
:3:hip_event.cpp            :367 : 159551320284 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320287 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x92be500 ) �[0m
:3:hip_event.cpp            :367 : 159551320290 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320293 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x92be600 ) �[0m
:3:hip_event.cpp            :367 : 159551320296 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320299 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x92be700 ) �[0m
:3:hip_event.cpp            :367 : 159551320302 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320305 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x92be800 ) �[0m
:3:hip_event.cpp            :367 : 159551320308 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320312 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x92be900 ) �[0m
:3:hip_event.cpp            :367 : 159551320315 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320318 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x93dcae0 ) �[0m
:3:hip_event.cpp            :367 : 159551320321 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320324 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x93dcbe0 ) �[0m
:3:hip_event.cpp            :367 : 159551320327 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320330 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x93dcce0 ) �[0m
:3:hip_event.cpp            :367 : 159551320333 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 
8000
: 159551320337 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x93dcde0 ) �[0m
:3:hip_event.cpp            :367 : 159551320340 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320343 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x93dcee0 ) �[0m
:3:hip_event.cpp            :367 : 159551320346 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320349 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x92688a0 ) �[0m
:3:hip_event.cpp            :367 : 159551320357 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_event.cpp            :347 : 159551320360 us: [pid:740193 tid: 0x778c75a53740] �[32m hipEventDestroy ( event:0x92689a0 ) �[0m
:3:hip_event.cpp            :367 : 159551320364 us: [pid:740193 tid: 0x778c75a53740] hipEventDestroy: Returned hipSuccess : 
:3:hip_stream.cpp           :394 : 159551320424 us: [pid:740193 tid: 0x778c75a53740] �[32m hipStreamDestroy ( stream:0x88d4840 ) �[0m
:4:rocdevice.cpp            :2387: 159551320438 us: [pid:740193 tid: 0x778c75a53740] Free hsa memory (nil)
:3:rocdevice.cpp            :3219: 159551320442 us: [pid:740193 tid: 0x778c75a53740] releaseQueue refCount:0x77843ce00000 (0)
:4:rocdevice.cpp            :2387: 159551320452 us: [pid:740193 tid: 0x778c75a53740] Free hsa memory 0x77843ca00000
:4:rocdevice.cpp            :2387: 159551321399 us: [pid:740193 tid: 0x778c75a53740] Free hsa memory 0x77843c400000
:3:hip_stream.cpp           :435 : 159551321431 us: [pid:740193 tid: 0x778c75a53740] hipStreamDestroy: Returned hipSuccess : 
:3:hip_stream.cpp           :394 : 159551321436 us: [pid:740193 tid: 0x778c75a53740] �[32m hipStreamDestroy ( stream:0x88f8f20 ) �[0m
:4:rocdevice.cpp            :2387: 159551321453 us: [pid:740193 tid: 0x778c75a53740] Free hsa memory (nil)
:3:rocdevice.cpp            :3219: 159551321458 us: [pid:740193 tid: 0x778c75a53740] releaseQueue refCount:0x77843c200000 (0)
:4:rocdevice.cpp            :2387: 159551321634 us: [pid:740193 tid: 0x778c75a53740] Free hsa memory 0x77843cb00000
:4:rocdevice.cpp            :2387: 159551322566 us: [pid:740193 tid: 0x778c75a53740] Free hsa memory 0x774c22000000
:3:hip_stream.cpp           :435 : 159551322594 us: [pid:740193 tid: 0x778c75a53740] hipStreamDestroy: Returned hipSuccess : 
:3:hip_context.cpp          :390 : 159551322599 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDevicePrimaryCtxRelease ( 0 ) �[0m
:3:hip_context.cpp          :396 : 159551322603 us: [pid:740193 tid: 0x778c75a53740] hipDevicePrimaryCtxRelease: Returned hipSuccess : 
c/runtime/src/iree/hal/drivers/hip/event_semaphore.c:786: ABORTED; the semaphore was aborted; while invoking native function hal.fence.await; while calling import; 
[ 0] bytecode module.conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g:92 repro.mlir:29:5; invoking function 'conv_2d_float32_forward_11x470x725x32_nhwc_224x1x1x32_fhwc_nhwf_1x1s_0x0p_1x1d_1g'
:3:hip_device_runtime.cpp   :641 : 159551322971 us: [pid:740193 tid: 0x778c75a53740] �[32m hipDeviceSynchronize (  ) �[0m
:3:hip_device_runtime.cpp   :645 : 159551322979 us: [pid:740193 tid: 0x778c75a53740] hipDeviceSynchronize: Returned hipSuccess : 
:4:rocdevice.cpp            :2387: 159551324086 us: [pid:740193 tid: 0x778c75a53740] Free hsa memory 0x778c75724000
:3:rocdevice.cpp            :283 : 159551324099 us: [pid:740193 tid: 0x778c75a53740] Deleting hardware queue 0x77843ce00000 with refCount 0
:3:rocdevice.cpp            :283 : 159551341452 us: [pid:740193 tid: 0x778c75a53740] Deleting hardware queue 0x77843c200000 with refCount 0
:4:rocdevice.cpp            :2387: 159551359666 us: [pid:740193 tid: 0x778c75a53740] Free hsa memory 0x77843e600000
:4:rocdevice.cpp            :2387: 159551361856 us: [pid:740193 tid: 0x778c75a53740] Free hsa memory 0x77843e400000
:4:rocdevice.cpp            :2387: 159551362886 us: [pid:740193 tid: 0x778c75a53740] Free hsa memory 0x77843e200000
:4:rocdevice.cpp            :2387: 159551363896 us: [pid:740193 tid: 0x778c75a53740] Free hsa memory 0x77843e000000
:4:rocdevice.cpp            :2387: 159551364900 us: [pid:740193 tid: 0x778c75a53740] Free hsa memory 0x77843de00000
:4:rocdevice.cpp            :2387: 159551365881 us: [pid:740193 tid: 0x778c75a53740] Free hsa memory 0x77843dc00000
:4:rocdevice.cpp            :2387: 159551366921 us: [pid:740193 tid: 0x778c75a53740] Free hsa memory 0x77843da00000
:4:rocdevice.cpp            :2387: 159551367908 us: [pid:740193 tid: 0x778c75a53740] Free hsa memory 0x77843d800000

@benvanik
Copy link
Collaborator
benvanik commented May 9, 2025 
:3:hip_module.cpp           :500 : 159551316598 us: [pid:740193 tid: 0x774c20ffd640] �[32m hipModuleLaunchKernel ( 0x0x92e3300, 7, 117133, 1, 256, 1, 1, 8960, stream:0x88d4840, 0x8ba8290, char array:<null> ) �[0m
:3:hip_module.cpp           :513 : 159551316605 us: [pid:740193 tid: 0x774c20ffd640] hipModuleLaunchKernel: Returned hipErrorInvalidValue :

ya launched bad

@MaheshRavishankar MaheshRavishankar mentioned this issue May 13, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug 🐞 Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Codegen][LLVMGPU] Optionally linearize the number of workgroups specified MaheshRavishankar/iree
2 participants
@benvanik @zjgarvey
2ACF
0