8000 WAMR upgrade by csegarragonz · Pull Request #734 · faasm/faasm · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

WAMR upgrade #734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 50 commits into from
Apr 4, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
e32b5a3
wamr: barebones mpi implmenetation to run lammps
csegarragonz Mar 21, 2023
0aafc27
wamr: more mpi functions working
csegarragonz Mar 22, 2023
bdc8be8
wamr: fix MPI_IN_PLACE detection to get MPI_Reduce to work properly
csegarragonz Mar 22, 2023
6a0c681
wamr: implement more mpi calls, mpi_cart_create not working
csegarragonz Mar 22, 2023
2f78cca
wamr: all mpi stubs for lammps in place
csegarragonz Mar 22, 2023
621b69c
wamr: mpi tests passing locally
csegarragonz Mar 22, 2023
fc5dbd9
nits: self-review cleanup
csegarragonz Mar 22, 2023
52e6359
mpi: add logging
csegarragonz Mar 22, 2023
39e3aad
wamr: unaligned write to make ubsan happy
csegarragonz Mar 22, 2023
aa635c7
wamr: make ubsan happy with mpi cart creation
csegarragonz Mar 23, 2023
8e8ef52
wamr: more ubsan fixes
csegarragonz Mar 23, 2023
43afe74
wamr fixes
csegarragonz Mar 27, 2023
984a128
cmake: latest wamr
csegarragonz Mar 27, 2023
b59dde1
wamr-mpi: fixes after rebase
csegarragonz Mar 28, 2023
f2e5214
nits: format
csegarragonz Mar 28, 2023
547982d
wamr: check the return val 8000 ue of wasm_runtime_module_malloc
csegarragonz Mar 29, 2023
e16c25b
wamr: lammps workinggit add src/wamr/
csegarragonz Mar 29, 2023
7ebdf88
nits
csegarragonz Mar 29, 2023
7c831f1
nit: run clang-format
csegarragonz Mar 29, 2023
19d5930
gh: bump code version
csegarragonz Mar 29, 2023
5125114
runner: remove unnecessary logging from local pool runner
csegarragonz Mar 30, 2023
5c36f74
wamr: trace logging for munmap
csegarragonz Mar 30, 2023
3bf0fc6
wamr: module cleanup and re-order prepare filesystem
csegarragonz Mar 30, 2023
dbdddb3
tests: add regression tests for lack of fd clearing
csegarragonz Mar 30, 2023
eabc039
tests: add fixture for fd tests
csegarragonz Mar 30, 2023
8d34fca
nits: run clang-format
csegarragonz Mar 30, 2023
73217d6
nits: self-review
csegarragonz Mar 30, 2023
df956ed
local pool runner: longer timeouts for long executions
csegarragonz Mar 30, 2023
5a170c9
nits: more self-review
csegarragonz Mar 30, 2023
55283cf
clang-format
csegarragonz Mar 30, 2023
681af88
nits: more self-review
csegarragonz Mar 30, 2023
5a2b525
enclave: stop using deprecated API call
csegarragonz Mar 30, 2023
858450a
wamr: update cmake reference
csegarragonz Mar 30, 2023
af147eb
cpp: bump tag version
csegarragonz Mar 30, 2023
94661aa
tests: fix disassemble test, which had changed after including the me…
csegarragonz Mar 31, 2023
dc75e2b
cmake: clean-up wamr's diff
csegarragonz Mar 31, 2023
c203496
k8s: manually bump the k8s-wamr versions after rebase
csegarragonz Mar 31, 2023
83aa7c4
wamr: move chaining into (old) funcs and refactor funcs to faasm
csegarragonz Apr 3, 2023
5a11817
migration: move to wasm
csegarragonz Apr 3, 2023
974c53b
wasm: move statement back to warn logging after memory leak has been …
csegarragonz Apr 3, 2023
92ad6d8
cpp: bump after merge
csegarragonz Apr 3, 2023
fa74c2a
wasm: remove todo after memory leak issue has been fixed
csegarragonz Apr 3, 2023
b2a3c3f
cpp: bump client after merge
csegarragonz Apr 3, 2023
e9660e4
wamr: fix warning
csegarragonz Apr 3, 2023
287e740
Merge branch 'wamr-migrate' into wamr-upgrade
csegarragonz Apr 3, 2023
9e3725b
nits: run clang-format after merge
csegarragonz Apr 3, 2023
be3d081
tests: fix disas test after mmap/munmap are not imported
csegarragonz Apr 3, 2023
692ba7e
wavm: remove commented out code
csegarragonz Apr 3, 2023
0bb6b1a
nit: run clang-format
csegarragonz Apr 3, 2023
6691075
wamr: squash to one-commit-diff with upstream main
csegarragonz Apr 4, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .env
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
FAASM_VERSION=0.9.5
FAASM_CLI_IMAGE=faasm.azurecr.io/cli:0.9.5
FAASM_WORKER_IMAGE=faasm.azurecr.io/worker:0.9.5
FAASM_VERSION=0.9.6
FAASM_CLI_IMAGE=faasm.azurecr.io/cli:0.9.6
FAASM_WORKER_IMAGE=faasm.azurecr.io/worker:0.9.6

CPP_VERSION=0.2.4
CPP_CLI_IMAGE=faasm.azurecr.io/cpp-sysroot:0.2.4
8000 CPP_VERSION=0.2.5
CPP_CLI_IMAGE=faasm.azurecr.io/cpp-sysroot:0.2.5

PYTHON_VERSION=0.2.5
PYTHON_CLI_IMAGE=faasm.azurecr.io/cpython:0.2.5
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/sgx_hw.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
runs-on: self-hosted
env:
VM_BASE_NAME: gha-sgx-hw-vm
FAASM_VERSION: 0.9.5
FAASM_VERSION: 0.9.6
steps:
- name: "Check out the experiment-base code"
uses: actions/checkout@v3
Expand Down
20 changes: 10 additions & 10 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
if: github.event.pull_request.draft == false
runs-on: ubuntu-latest
container:
image: faasm.azurecr.io/cli:0.9.5
image: faasm.azurecr.io/cli:0.9.6
credentials:
username: ${{ secrets.ACR_SERVICE_PRINCIPAL_ID }}
password: ${{ secrets.ACR_SERVICE_PRINCIPAL_PASSWORD }}
Expand Down Expand Up @@ -52,7 +52,7 @@ jobs:
if: github.event.pull_request.draft == false
runs-on: ubuntu-latest
container:
image: faasm.azurecr.io/cli:0.9.5
image: faasm.azurecr.io/cli:0.9.6
credentials:
username: ${{ secrets.ACR_SERVICE_PRINCIPAL_ID }}
password: ${{ secrets.ACR_SERVICE_PRINCIPAL_PASSWORD }}
Expand All @@ -68,7 +68,7 @@ jobs:
if: github.event.pull_request.draft == false
runs-on: ubuntu-latest
container:
image: faasm.azurecr.io/cpp-sysroot:0.2.4
image: faasm.azurecr.io/cpp-sysroot:0.2.5
credentials:
username: ${{ secrets.ACR_SERVICE_PRINCIPAL_ID }}
password: ${{ secrets.ACR_SERVICE_PRINCIPAL_PASSWORD }}
Expand Down Expand Up @@ -131,7 +131,7 @@ jobs:
if: github.event.pull_request.draft == false
runs-on: ubuntu-latest
container:
image: faasm.azurecr.io/cli:0.9.5
image: faasm.azurecr.io/cli:0.9.6
credentials:
username: ${{ secrets.ACR_SERVICE_PRINCIPAL_ID }}
password: ${{ secrets.ACR_SERVICE_PRINCIPAL_PASSWORD }}
Expand Down Expand Up @@ -168,18 +168,18 @@ jobs:
TSAN_OPTIONS: "history_size=0 halt_on_error=1 suppressions=./thread-sanitizer-ignorelist.txt flush_memory_ms=5000"
UBSAN_OPTIONS: "print_stacktrace=1:halt_on_error=1:suppressions=./ub-sanitizer-ignorelist.txt"
container:
image: faasm.azurecr.io/cli:0.9.5
image: faasm.azurecr.io/cli:0.9.6
credentials:
username: ${{ secrets.ACR_SERVICE_PRINCIPAL_ID }}
password: ${{ secrets.ACR_SERVICE_PRINCIPAL_PASSWORD }}
services:
redis:
image: faasm.azurecr.io/redis:0.9.5
image: faasm.azurecr.io/redis:0.9.6
credentials:
username: ${{ secrets.ACR_SERVICE_PRINCIPAL_ID }}
password: ${{ secrets.ACR_SERVICE_PRINCIPAL_PASSWORD }}
minio:
image: faasm.azurecr.io/minio:0.9.5
image: faasm.azurecr.io/minio:0.9.6
credentials:
username: ${{ secrets.ACR_SERVICE_PRINCIPAL_ID }}
password: ${{ secrets.ACR_SERVICE_PRINCIPAL_PASSWORD }}
Expand Down Expand Up @@ -257,18 +257,18 @@ jobs:
REDIS_QUEUE_HOST: redis
REDIS_STATE_HOST: redis
container:
image: faasm.azurecr.io/cli-sgx-sim:0.9.5
image: faasm.azurecr.io/cli-sgx-sim:0.9.6
credentials:
username: ${{ secrets.ACR_SERVICE_PRINCIPAL_ID }}
password: ${{ secrets.ACR_SERVICE_PRINCIPAL_PASSWORD }}
services:
redis:
image: faasm.azurecr.io/redis:0.9.5
image: faasm.azurecr.io/redis:0.9.6
credentials:
username: ${{ secrets.ACR_SERVICE_PRINCIPAL_ID }}
password: ${{ secrets.ACR_SERVICE_PRINCIPAL_PASSWORD }}
minio:
image: faasm.azurecr.io/minio:0.9.5
image: faasm.azurecr.io/minio:0.9.6
credentials:
username: ${{ secrets.ACR_SERVICE_PRINCIPAL_ID }}
password: ${{ secrets.ACR_SERVICE_PRINCIPAL_PASSWORD }}
Expand Down
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0.9.5
0.9.6
2 changes: 1 addition & 1 deletion cmake/ExternalProjects.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ FetchContent_Declare(wavm_ext

FetchContent_Declare(wamr_ext
GIT_REPOSITORY "https://github.com/faasm/wasm-micro-runtime"
GIT_TAG "a31e5a4fa299c4f8384f40e157b0a928ad0bda1b"
GIT_TAG "5e9dc3c7eb33167389d99b7e5851dc55b5911d33"
)

# WAMR and WAVM both link to LLVM
Expand Down
2 changes: 1 addition & 1 deletion deploy/k8s-common/minio.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ metadata:
spec:
containers:
- name: minio-main
image: faasm.azurecr.io/minio:0.9.5
image: faasm.azurecr.io/minio:0.9.6
env:
- name: MINIO_ROOT_USER
value: "minio"
Expand Down
4 changes: 2 additions & 2 deletions deploy/k8s-common/redis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ metadata:
spec:
containers:
- name: master
image: faasm.azurecr.io/redis:0.9.5
image: faasm.azurecr.io/redis:0.9.6
ports:
- containerPort: 6379

Expand All @@ -28,7 +28,7 @@ metadata:
spec:
containers:
- name: master
image: faasm.azurecr.io/redis:0.9.5
image: faasm.azurecr.io/redis:0.9.6
ports:
- containerPort: 6379

Expand Down
2 changes: 1 addition & 1 deletion deploy/k8s-sgx/upload.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ metadata:
spec:
containers:
- name: upload
image: faasm.azurecr.io/upload:0.9.5
image: faasm.azurecr.io/upload:0.9.6
ports:
- containerPort: 8002
- containerPort: 5000
Expand Down
2 changes: 1 addition & 1 deletion deploy/k8s-sgx/worker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ spec:
weight: 100

containers:
- image: faasm.azurecr.io/worker-sgx:0.9.5
- image: faasm.azurecr.io/worker-sgx:0.9.6
name: faasm-worker
ports:
- containerPort: 8080
Expand Down
2 changes: 1 addition & 1 deletion deploy/k8s-wamr/upload.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ metadata:
spec:
containers:
- name: upload
image: faasm.azurecr.io/upload:0.9.5
image: faasm.azurecr.io/upload:0.9.6
ports:
- containerPort: 8002
- containerPort: 5000
Expand Down
2 changes: 1 addition & 1 deletion deploy/k8s-wamr/worker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ spec:
weight: 100

containers:
- image: faasm.azurecr.io/worker:0.9.5
- image: faasm.azurecr.io/worker:0.9.6
name: faasm-worker
ports:
- containerPort: 8080
Expand Down
2 changes: 1 addition & 1 deletion deploy/k8s/upload.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ metadata:
spec:
containers:
- name: upload
image: faasm.azurecr.io/upload:0.9.5
image: faasm.azurecr.io/upload:0.9.6
ports:
- containerPort: 8002
- containerPort: 5000
Expand Down
2 changes: 1 addition & 1 deletion deploy/k8s/worker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ spec:
weight: 100

containers:
- image: faasm.azurecr.io/worker:0.9.5
- image: faasm.azurecr.io/worker:0.9.6
name: faasm-worker
ports:
- containerPort: 8080
Expand Down
30 changes: 26 additions & 4 deletions include/wamr/WAMRModuleMixin.h
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

#include <wasm_export.h>

#include <stdexcept>
#include <string>
#include <vector>

Expand All @@ -28,12 +29,15 @@ struct WAMRModuleMixin

// Validate that a memory range defined by a pointer and a size is a valid
// offset in the module's WASM linear memory.
// bool validateNativePointer(void* nativePtr, size_t size);
bool validateNativePointer(void* nativePtr, int size)
void validateNativePointer(void* nativePtr, int size)
{
auto moduleInstance = this->underlying().getModuleInstance();
return wasm_runtime_validate_native_addr(
moduleInstance, nativePtr, size);
bool success =
wasm_runtime_validate_native_addr(moduleInstance, nativePtr, size);

if (!success) {
throw std::runtime_error("Failed validating native pointer!");
}
}

void* wasmOffsetToNativePointer(uint32_t wasmOffset)
Expand All @@ -50,6 +54,24 @@ struct WAMRModuleMixin
return wasm_runtime_addr_native_to_app(moduleInstance, nativePtr);
}

// Allocate memory in the WASM's module heap (inside the linear memory).
// Returns the WASM offset of the newly allocated memory if succesful, 0
// otherwise. If succesful, populate the nativePtr variable with the
// native pointer to access the returned offset
uint32_t wasmModuleMalloc(size_t size, void** nativePtr)
{
auto moduleInstance = this->underlying().getModuleInstance();
uint32_t wasmOffset =
wasm_runtime_module_malloc(moduleInstance, size, nativePtr);
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This essentially calls the malloc symbol in wasi-libc. Hence why we need to export it here.


if (wasmOffset == 0 || nativePtr == nullptr) {
throw std::runtime_error(
"Failed malloc-ing memory in WASM module!");
}

return wasmOffset;
}

// Helper function to write a string array to a buffer in the WASM linear
// memory, and record the offsets where each new string begins (note that
// in WASM this strings are now interpreted as char pointers).
Expand Down
2 changes: 0 additions & 2 deletions include/wamr/native.h
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,6 @@
namespace wasm {
void initialiseWAMRNatives();

uint32_t getFaasmChainingApi(NativeSymbol** nativeSymbols);

uint32_t getFaasmDynlinkApi(NativeSymbol** nativeSymbols);

uint32_t getFaasmEnvApi(NativeSymbol** nativeSymbols);
Expand Down
6 changes: 6 additions & 0 deletions include/wasm/migration.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#pragma once

namespace wasm {
void doMigrationPoint(int32_t entrypointFuncWasmOffset,
const std::string& entrypointFuncArg);
}
26 changes: 15 additions & 11 deletions src/enclave/inside/EnclaveWasmModule.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -60,26 +60,30 @@ bool EnclaveWasmModule::loadWasm(void* wasmOpCodePtr, uint32_t wasmOpCodeSize)

bool EnclaveWasmModule::callFunction(uint32_t argcIn, char** argvIn)
{
prepareArgcArgv(argcIn, argvIn);

WASMExecEnv* execEnv = wasm_runtime_get_exec_env_singleton(moduleInstance);
if (execEnv == nullptr) {
ocallLogError("Failed to create WAMR exec env");
throw std::runtime_error("Failed to create WAMR exec env");
}

WASMFunctionInstanceCommon* func =
wasm_runtime_lookup_function(moduleInstance, WASM_ENTRY_FUNC, nullptr);

prepareArgcArgv(argcIn, argvIn);
if (func == nullptr) {
ocallLogError("Did not find named WASM function");
throw std::runtime_error("Did not find named wasm function");
}

// Set dummy argv to capture return value
std::vector<uint32_t> argv = { 0 };

bool success =
aot_create_exec_env_and_call_function((AOTModuleInstance*)moduleInstance,
(AOTFunctionInstance*)func,
0x0,
argv.data());
bool success = wasm_runtime_call_wasm(execEnv, func, 0, argv.data());
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only reason I change this here is because aot_create_exec_env_and_call_function is not even exposed as an API function anymore, so this file would not compile.

uint32_t returnValue = argv[0];

if (success) {
ocallLogDebug("Success calling WASM function");
} else {
std::string errorMessage(
((AOTModuleInstance*)moduleInstance)->cur_exception);
// TODO - better logging
std::string errorMessage(wasm_runtime_get_exception(moduleInstance));
std::string errorText =
"Caught WASM runtime exception: " + errorMessage;
ocallLogError(errorText.c_str());
Expand Down
13 changes: 8 additions & 5 deletions src/runner/local_pool_runner.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,9 @@ int doRunner(int argc, char* argv[])

if (vm.count("input-data")) {
msg.set_inputdata(vm["input-data"].as<std::string>());
SPDLOG_INFO("Adding input data: {}",
vm["input-data"].as<std::string>());
}
if (vm.count("cmdline")) {
msg.set_cmdline(vm["cmdline"].as<std::string>());
SPDLOG_INFO("Adding command line arguments: {}",
vm["cmdline"].as<std::string>());
}

faabric::scheduler::Scheduler& sch = faabric::scheduler::getScheduler();
Expand All @@ -32,7 +28,7 @@ int doRunner(int argc, char* argv[])
usleep(1000 * 500);

for (const auto& m : req->messages()) {
faabric::Message result = sch.getFunctionResult(m.id(), 20000);
faabric::Message result = sch.getFunctionResult(m.id(), 20000 * 100);
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the local pool runner it makes sense to have longer timeouts, as we may want to run some longer running functions for debug purposes.

As a reminder, the local pool runner is a pool_runner that can be invoked from the command line. For example, the following runs a LAMMPS simulation without requiring any changes to faasm (i.e. any new lammps_runner etc):

 WASM_VM=wamr inv run.pool lammps main --cmdline '-in faasm://lammps-data/in.controller.wall'

if (result.returnvalue() != 0) {
SPDLOG_ERROR("Message ({}) returned error code: {}",
m.id(),
Expand All @@ -53,6 +49,13 @@ int main(int argc, char* argv[])
sch.shutdown();
sch.addHostToGlobalSet();

// Set timeout to ensure longer functions can finish
faabric::util::SystemConfig& conf = faabric::util::getSystemConfig();
conf::FaasmConfig& faasmConf = conf::getFaasmConfig();
conf.boundTimeout = 120000 * 100;
conf.globalMessageTimeout = 120000 * 100;
faasmConf.chainedCallTimeout = 120000 * 100;

// WARNING: All 0MQ-related operations must take place in a self-contined
// scope to ensure all sockets are destructed before closing the context.
{
Expand Down
4 changes: 2 additions & 2 deletions src/wamr/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ set(WAMR_BUILD_SPEC_TEST 0)
add_definitions(-DWAMR_FAASM=1)

# Set AOT mode and JIT for code generation
set(WAMR_BUILD_INTERPRETER 1)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was set by default implicitly through setting WAMR_BUILD_AOT, but I prefer to make it explicit.

set(WAMR_BUILD_AOT 1)
set(WAMR_BUILD_JIT 1)
set(WAMR_BUILD_LAZY_JIT 0)
Expand Down Expand Up @@ -74,12 +75,11 @@ llvm_map_components_to_libnames(
# Link everything together
faasm_private_lib(wamrmodule
WAMRWasmModule.cpp
chaining.cpp
codegen.cpp
dynlink.cpp
env.cpp
faasm.cpp
filesystem.cpp
funcs.cpp
memory.cpp
mpi.cpp
native.cpp
Expand Down
Loading
0