Rust Compilation for Lind-Wasm Support

[Yaxuan-w]

To compile rustc into lind-wasm, our first step is to ensure regular Rust programs can run successfully within the lind-wasm runtime. This issue is to track the progress of compiling Rust code into the lind-wasm.

Current Approach

Because Rust uses the LLVM backend, and lind-wasm relies heavily on custom build settings during both object generation and linking, we use similar strategies in Rust:

Thread support during compilation:

Just like in the C compilation process for lind-wasm, we need to enable -pthread at the compilation stage to ensure correct threading behavior by wasm32-wasip1-threads. In rust the way to do so, is to enable the '+atomics' feature in unison with the wasm32-wasip1-threads target, which we have configured.

Custom memory and stack behavior during linking:

At the linking stage, lind-wasm requires specific flags to manage memory and stack behavior in wasmtime:

• --export-memory
• --import-memory
• --max-memory
• --export=__stack_pointer
• --export=__stack_low

Though import-memory and export-memory are somewhat redundant, --import-memory tells the linker that the wasm module will import memory from the host (lind-wasm runtime) and --export memory tells the linker that the wasm module must make it's memory accessible to the host. For our tests, we will be keeping both flags as LLVM 16 allows and requires both for wasi-threads. See issue: WebAssembly/WASI#502

In addition, we will also run unit tests without the --import-memory to see if it is truly required for successful compilation and execution with wasi and wasi-threads support.

To implement this in Rust, we configure those in .cargo/config.toml:

• A target of wasm32-wasip1-threads
• A custom linker that invokes our modified wasm-ld with lind-wasm-specific flags
• Include the tokio crate in the list of dependencies with the following features enabled to test async functions: "rt", "time", "macros", "io-util", "sync"

Current Progress

With help from Sankalp @ssannkkallpp , we successfully compiled and ran:

• A Rust Hello World program
• A synchronized write test
• Simple rust programs with multiple I/O operations and computations performed in sequence without blocking the runtime as a whole
• Simple asynchronous rust programs with tokio

These ran correctly under lind-wasm without errors and the asynchronous processes run without blocking the runtime.

--import-memory Conflict

We noticed that using both --import-memory and --export-memory leads to a conflict. Dropping the --import-memory flag doesn't appear to impact current functionality. Rust tests run fine without it. C programs also behave correctly without --import-memory.

Further discussion is needed to determine whether --import-memory is fundamentally incompatible with our setup or if it might be required in specific cases.

Next Steps

@ssannkkallpp will add more Rust-based test cases to improve coverage. Once stabilized, a PR will be submitted with working examples and build changes. We also need to discuss how to integrate Rust compilation into our current CI/CD pipeline.

@ssannkkallpp will be experimenting with more C programs and tests by compiling for wasm32 without the --import-memory flag to ensure no regression.

[ssannkkallpp]

The current status for unit testing rust code that is cross-compiled to wasm32-wasip1-threads is as follows.

1. Updated the wasmtestreport.py/lind_compile scripts to allow support for native and cross-compilation of .rs test files.

1. The script creates a temporary Cargo.toml in a temp_rust_build directory for each .rs test file, using standard Rust toolchain without custom targets or sysroot flags. It runs cargo build --release to generate native binaries for comparison testing.

2. The script uses the updated lind_compile script which creates a separate temporary environment with a custom .cargo/config.toml containing the above linker flags and the wasm32-wasip1-threads target.

2. Have generated basic unit test files in rust for memory and file tests by porting existing C test cases to Rust, maintaining the same deterministic/non-deterministic categorization. Each test uses Rust's standard library APIs (like File, Write, memcpy, etc.). Currently, a lot of test cases that are present in skip_test_cases.txt just hold placeholder code since the test report generation does not run them (even for C code), but this will be sequentially updated to complete out the full rust unit test suite.

3. In addition to the linker flags mentioned in the original issue, the following key linker flag is also passed as the sysroot:
   --export=/home/lind/lind-wasm/src/glibc/sysroot

@Yaxuan-w Though all unskipped unit tests are compiling successfully (with memory and file tests even succeeding), when invoking certain calls such as from std::net::{UdpSocket, Ipv4Addr, SocketAddr} for config tests, I am encountering the following error:
Error: Error { kind: Unsupported, message: "operation not supported on this platform" }

My best guess is I have missed some error that occurs during the linking phase to the correct sysroot. However, interesting point to note is most std rust operations seem to compile and execute correctly in lind-wasm with/without our custom sysroot.