YAMAMOTO Takashi [Mon, 21 Aug 2023 17:21:37 +0000 (02:21 +0900)]
Fix recursive mutex (#433)
the robust mutex logic in musl seems to assume that
the bit 29 of TIDs is always zero for some reasons.
from https://git.musl-libc.org/cgit/musl/commit/?id=099b89d3840c30d7dd962e18668c2e6d39f0c626
> note that the kernel ABI also reserves bit 29
> not to appear in any tid,
i'm not sure if the assumption is true or not, given that
FUTEX_TID_MASK is 0x3fffffff.
anyway, when using non-default type of mutex like recursive mutex,
it causes problems as we actually use TID 0x3fffffff for the main thread.
as we don't support robust mutex anyway, this commit simply
comments out the problematic condition.
Marcin Kolny [Thu, 13 Jul 2023 16:18:08 +0000 (17:18 +0100)]
Add definitions for PF_INET, PF_INET6 and PF_UNSPEC (#426)
Given there are already AF_* definitions, and they are (now) essentially
synonyms, we add those definitions to enable compilation of code that
already use PF_* macros.
Moritz Sichert [Tue, 11 Jul 2023 22:33:07 +0000 (00:33 +0200)]
Acquire the global lock before initializing malloc (#410)
In a multi-threaded execution we need to make sure that only exactly one
thread initializes malloc. The function try_init_allocator() can't
easily be made thread-safe, so just move the call to
try_init_allocator() inside the block that holds the lock.
This commit effectively drops the support of older wasm-ld. (LLVM <15.0.7).
We have two relevant use cases:
* `memory.grow` use outside of malloc
(eg. used by polyfill preview1 binaries)
* `--init-memory` to somehow preallocate heap
(eg. avoid dynamic allocations, especially on small environments)
While https://github.com/WebAssembly/wasi-libc/pull/377
fixed the former, it broke the latter if you are using
an older LLVM, which doesn't provide the `__heap_end` symbol,
to link your module.
As we couldn't come up with a solution which satisfies all parties,
this commit simply makes it require new enough LLVM which provides
`__heap_end`. After all, a link-time failure is more friendly to users
than failing later in a subtle way.
Andrew Brown [Fri, 23 Jun 2023 20:24:08 +0000 (13:24 -0700)]
Improve `README.md` (#425)
This changes the front-page documentation to:
- use `wasi-libc` instead of "WASI Libc"
- explain how to build the pthreads-enabled `wasm32-wasi-threads` target
Dan Gohman [Wed, 3 May 2023 19:44:21 +0000 (12:44 -0700)]
Convert preopen initialization to be lazy. (#408)
* Convert preopen initialization to be lazy.
Insteead of eagerly initializing the preopens in a static constructor,
perform preopen initialization the first time it's needed, or before a
close or a renumber which might disrupt the file descriptor space.
And, use a weak symbol with a stub function for use by `close` or
`fd_renumber`, we that they can trigger preopen initialization only
if it's actually needed.
This way, if a program doesn't contain any calls to any function that
needs preopens, it can avoid linking in the preopen initialization code.
And if it contains calls but doesn't execute them at runtime, it can
avoid executing the preopen intiailization code.
A downside here is that this may cause problems for users that call
`__wasi_fd_close` or `__wasi_fd_renumber` directly and close over
overwrite some preopens before libc has a chance to scan them. To
partially address this, this PR does add a declaration for
`__wasilibc_populate_preopens` to <wasi/libc.h> so that users can call
it manually if they need to.
* Fix calling `internal_register_preopened_fd` with the lock held.
Factor out the lock acquisition from the implementation of
`internal_register_preopened_fd` so that we can call it from
`__wasilibc_populate_preopens` with the lock held.
Alex Crichton [Wed, 5 Apr 2023 19:13:39 +0000 (14:13 -0500)]
Fix debug build's `predefined-macros.txt` (#407)
This commit fixes the ability to build `wasi-libc` with `-g` options and
possibly without `-O2` options as well. I've found this useful when
debugging issues as historically that the build fails when `-g` is
passed or optimizations are removed due to the checks against these
expectation files. This commit adds more filters to the list of macros
to ensure that optimization/debug related ones are all removed from the
expectation lists.
Sam Clegg [Thu, 2 Mar 2023 01:14:42 +0000 (17:14 -0800)]
Avoid using absolute pathnames in Makefile. NFC (#400)
This makes the output of the build a lot more concise and easy to read.
The only real change here is to build each of the crt1 startup files
individually instead to trying to build them all in a single clang
invocation (that latter doesn't allow for -o to be specified which is
a pretty severe limitation, so its best avoided anyway).
It also reduces the size of the `ar` command line for libc itself from
78017 to 43609 (on my machine), which sadly is still tool long for win32
I believe.
Cheng Shao [Mon, 30 Jan 2023 12:30:02 +0000 (13:30 +0100)]
Add a check to reactor modules to ensure _initialize is only called once (#388)
Calling _initialize multiple times is undefined behavior, since the
ctors are not guaranteed to be idempotent. We should have this safety
check which is similar to #329.
Andrew Brown [Fri, 13 Jan 2023 02:56:52 +0000 (18:56 -0800)]
threads: change `wasm32-wasi-pthread` to `wasm32-wasi-threads` (#381)
* Change `wasm32-wasi-pthread` to `wasm32-wasi-threads`
After some thought, I think that we should rename the `THREAD_MODEL=posix` build to avoid confusion. Why? Though in this project the use of this target does involve pthreads, it will not be so in other standard libraries or languages (see, e.g., https://github.com/rust-lang/compiler-team/issues/574). I think it would be preferable to emphasize the "threads" Wasm-level proposal and the "wasi-threads" proposal rather than the specific details of which threading API is being exposed.
* fix: rename the `expected` output directory as well
Alex Crichton [Mon, 9 Jan 2023 16:33:05 +0000 (10:33 -0600)]
Don't use sbrk(0) to determine the initial heap size (#377)
* Don't use sbrk(0) to determine the initial heap size
This commit changes the `try_init_allocator` function as part of
dlmalloc to not use `sbrk(0)` to determine the initial heap size. The
purpose of this function is to use the extra memory at the end of linear
memory for the initial allocation heap before `memory.grow` is used to
allocate more memory. To learn the extent of this region the code
previously would use `sbrk(0)` to find the current size of linear
memory. This does not work, however, when other systems have called
`memory.grow` before this function is called. For example if another
allocator is used or if another component of a wasm binary grows memory
for its own purposes then that memory will be incorrectly claimed to be
owned by dlmalloc.
Instead this commit rounds up the `__heap_base` address to the nearest
page size, since that must be allocatable. Otherwise anything above this
rounded address is assumed to be used by something else, even if it's
addressable.
Dan Gohman [Wed, 21 Dec 2022 23:47:53 +0000 (15:47 -0800)]
Use `ENOENT` rather than `ENOTCAPABLE` for missing preopens. (#370)
When a user calls `open` with a path that does not have a corresponding
preopen, set errno to `ENOENT` rather than `ENOTCAPABLE`. This
conceptually represents an attempt to open a path which has not been
provided within the sandbox, so it's more accurately represented as
"not present" rather than "insufficient capabilities".
YAMAMOTO Takashi [Wed, 21 Dec 2022 01:24:09 +0000 (10:24 +0900)]
Disable pthread_exit for now (#366)
The current wasi-threads has no thread-exit functionality.
Thus it isn't straightforward to implement pthread_exit
without leaking thread context. This commit simply disables
pthread_exit for now.
Also, instead of abusing `wasi_proc_exit` for thread exit,
make `wasi_thread_start` return.
Note: `wasi_proc_exit` is supposed to terminate all threads
in the "process", not only the calling thread.
Note: Depending on the conclusion of the discussion about
`wasi_thread_exit`, we might revisit this change later.
Marcin Kolny [Mon, 19 Dec 2022 12:18:19 +0000 (12:18 +0000)]
threads: Retrieve default stack size from __heap_base/__data_end (#350)
When compiling with `-z stack-size` flag, only the main thread's stack
size is set to the specified value and other threads use musl's default value.
That's inconsistent with LLD's `-Wl,-stack_size`.
I think we can make it similar to MUSL's behavior, where thread's stack
size can be set via `PT_GNU_STACK` program header (via `-Wl,-z,stack-size`
flag).
Configuring stack size through `pthread_attr_t` still work as expected and
overrides the defaults ([pthread_create.c](https://github.com/WebAssembly/wasi-libc/blob/be1ffd6a9eba1704085987482557c2a32724227f/libc-top-half/musl/src/thread/pthread_create.c#L362))
default settings.
Andrew Brown [Tue, 13 Dec 2022 17:19:32 +0000 (09:19 -0800)]
threads: enable access to `pthread_barrier_*` functions (#358)
In building some `libc-test` tests, I found these functions were not
compiled in. This change adds `pthread_barrier_init`,
`pthread_barrier_wait`, and `pthread_barrier_destroy` to the
`THREAD_MODEL=posix` build. As has been done with previous pthreads PRs,
this PR skips any inter-process locking by removing any calls to
`__vm_lock` and friends. If in the future WASI gains the "process"
concept, then these locations (and the pre-existing ones) will need to
be modified.
Andrew Brown [Thu, 8 Dec 2022 22:55:42 +0000 (14:55 -0800)]
threads: enable access to `pthread_attr_get` functions (#357)
The pthreads API exposes functions for querying the attributes of a
thread. This change allows these functions to be compiled in the
`THREAD_MODEL=posix` build. Some functions are skipped (and documented);
they can be added if/when needed. This change is motivated by a
`libc-test` test that uses these functions.
__wasi_thread_spawn: stop truncating the return value (#353)
as __wasi_errno_t is uint16_t, with the current coding,
__pthread_create will never see negative return values from
wasi:thread_spawn.
eg. (int)(uint16_t)-1 == 65535.
Mike Hommey [Wed, 7 Dec 2022 16:27:06 +0000 (01:27 +0900)]
Adjust Makefile for LLVM trunk (16) as of 2022-11-08 (#344)
https://github.com/llvm/llvm-project/commit/1e4e2433bcd1a0296ef1043c462252f0d087d90c
enabled sign-ext and mutable-globals by default, which adds
corresponding __wasm_-prefixed #defines.
https://github.com/llvm/llvm-project/commit/9e956995db1fc7e792e3dfb3a465a52626195557
changed the definition of __GNUC_VA_LIST to match that of GCC headers,
leaving it without a value.
Dan Gohman [Tue, 6 Dec 2022 17:16:45 +0000 (09:16 -0800)]
Fix logic errors in the zero-inode path. (#352)
I've now tested the zero-inode path on a Wasm engine specially-modified
to have `fd_readdir` set inode numbers to zero. Fix two bugs this turned up:
- Increment `buffer_processed`, as noticed by @yamt
- Don't do an `fstatat` on "..", because that references a path outside
of the directory, which gets a permission-denied error.
Andrew Brown [Sat, 3 Dec 2022 02:27:42 +0000 (18:27 -0800)]
test: run a subset of tests using `libc-test` (#346)
* test: run a subset of tests from `libc-test`
This change introduces a `test` directory that retrieves the `libc-test`
suite, compiles a subset of the tests using `wasi-libc`, and runs them
with Wasmtime.
* ci: run tests during CI
This change includes some fixups to the filesystem to place Clang's
runtime library for `wasm32-wasi` in the right location. Note that this
CI action is limited to a single OS--Linux.
Dan Gohman [Fri, 2 Dec 2022 01:44:22 +0000 (17:44 -0800)]
If `fd_readdir` returns a zero inode, call `fstatat` to get the inode value. (#345)
* If `fd_readdir` returns a zero inode, call `fstatat` to get the inode value.
On some systems, `fd_readdir` may not implement the `d_ino` field and
may set it to zero. When this happens, have wasi-libc call `fstatat` to
get the inode number.
See the discussion in
https://github.com/WebAssembly/wasi-filesystem/issues/65 for details.
* Update the `d_type` field too, in case it changes.
Dan Gohman [Wed, 30 Nov 2022 19:17:08 +0000 (11:17 -0800)]
Install libtinfo5 on ubuntu. (#349)
We download LLVM releases built for Ubuntu 18 because LLVM doesn't
always have builds for different versions, but the builds we use
depend on libtinfo5 which isn't installed on Ubuntu 20 by default.
So install it.
Dan Gohman [Mon, 28 Nov 2022 21:50:04 +0000 (13:50 -0800)]
Define an `__errno_location` function. (#347)
This function returns the address of `errno`, which makes it easier to
access from non-C languages since `errno` is a thread-local variable
which requires a special ABI.
Dan Gohman [Fri, 14 Oct 2022 23:09:55 +0000 (16:09 -0700)]
Port emmalloc to wasi-libc.
- Avoid using Emscripten-specific functions
- Avoid using a constructor.
- Add support for allocating memory at `__heap_base`.
- Adjust the max-align value.
- Disable functions that wasi-libc doesn't currently publish.
- Add `__libc_` aliases.
Dan Gohman [Fri, 14 Oct 2022 00:58:14 +0000 (17:58 -0700)]
Add a check to command modules to ensure that they're only started once. (#329)
* Add a check to command modules to ensure that they're only started once.
Wasm command modules should only be called once per instance, because
the programming model doesn't leave linear memory in a reusable state
when the program exits. As use cases arise for loading wasm modules in
environments that want to treat them like reactors, add a safety check
to ensure that command modules are used according to their
expectations.
Andrew Brown [Tue, 4 Oct 2022 14:21:18 +0000 (07:21 -0700)]
threads: implement `pthread_create` (#325)
* threads: implement `pthread_create`
As described in the [`wasi-threads`] proposal, this change implements
`pthread_create` using the new `wasi_thread_spawn(void *arg)` API. As
described there, `wasi-libc` exports the thread entry point with the
expected name, `wasi_thread_start`, and then unwraps the passed argument
`struct` to invoke the user function with the user argument `struct`.
Previously, the TID was only passed to the child thread entry point; the
parent thread was forced to wait until the child thread set the TID in
the pthread structure. With this change, the TID will be passed not only
to the child thread but also returned to the parent thread, so that
either side can make progress. The `i32.store` becomes an
`i32.atomic.store` to avoid concurrent writes.
Dan Gohman [Fri, 23 Sep 2022 18:46:59 +0000 (11:46 -0700)]
Don't run static constructors on arbitrary user exports.
Previously, "new-style commmands" considered every user-defined
export to be a potential command entrypoint, so wasi-libc and wasm-ld
cooperated to run the user's static constructors on each entrypoint.
This form of new-style command turned out not to be useful, and it
interferes with some use cases, so disable it.
This is done by making an explicit call to `__wasm_call_ctors`, which
tells wasm-ld that it shouldn't synthesize any calls to
`__wasm_call_ctors` on its own.
Marcin Kolny [Mon, 19 Sep 2022 20:53:39 +0000 (21:53 +0100)]
make `__get_tp()` a static function (#327)
I'm not sure if the function really has to be exported. If so, we should
probably move it to a separate compilation unit, otherwise it will be defined
multiple times (e.g. in `strerror.o` and `__lctrans.o`) causing linker errors.
However, I don't see a reason (at least for now) to export this function,
therefore making it static in this PR.
Andrew Brown [Mon, 22 Aug 2022 21:22:09 +0000 (14:22 -0700)]
threads: implement support for unnamed semaphores (#316)
[POSIX semaphores] come in two forms: named and unnamed. Roughly, named
semaphores use files to implement locking across processes; unnamed
semaphores use a shared memory region to implement locking across
threads in the same process. Since WASI currently has no process concept
(and it is relatively unclear how to map the WASI files as shared
memory), only the unnamed semaphores are supported by this changed. This
means that `sem_open`, `sem_close`, and `sem_unlink` will not available
to programs compiled with a threads-enabled `wasi-libc`.
Andrew Brown [Mon, 22 Aug 2022 15:39:44 +0000 (08:39 -0700)]
threads: implement support for pthread mutexes (#315)
This change adds pthread's mutex support to the `THREAD_MODEL=posix`
build of wasi-libc. Some less-common features are unsupported and
documented here:
- mutex robust lists are disabled and their use will return a runtime
error; currently WASI does not support the concept of multiple
processes so maintaining robust mutexes across processes does not yet
make sense in wasi-libc
- timed locks with priority inheritance (PI) are disabled and will act
as any other mutex; this feature is related to task priorities which
is not yet relevant in the WASI ecosystem (see [priority-inheritance
futexes](https://man7.org/linux/man-pages/man2/futex.2.html))
- thread cancellation is ignored; this feature is difficult to support
and @sunfishcode would likely want to discuss this before adding it at
some later time.
Andrew Brown [Tue, 9 Aug 2022 15:08:37 +0000 (08:08 -0700)]
Fix `make THREAD_MODEL=posix` (#311)
* Fixes for the THREAD_MODEL=posix build
* Fix expected symbols from previous commit
* Enable `lock` in `random.c` when threads are enabled
This uses the `_REENTRANT` definition to indicate when the `lock` should
be available.
* Disable `aio.h` when compiling for threads
In talking to @sunfishcode about `aio.h`, this functionality is not yet
a primary concern (it was already disabled in the default,
single-threaded mode). Additionally, this change adds expectation lines
for the new symbols/includes added and removed by `pthread.h`.
This change was reached by running:
```console
$ git diff --no-index expected/wasm32-wasi sysroot/share/wasm32-wasi > patch.diff
# replace "sysroot/share" with "expected" in `patch.diff`
$ git apply patch.diff --reject
# manually fix any rejections
```
* Specify the TLS model until LLVM 15 is released
The `-ftls-model` configuration can be removed once https://reviews.llvm.org/D130053 makes its way into an upstream release.
* Rename `__wasi_libc_pthread_self` to `__wasilibc_pthread_self`
The symbol is still undefined, though.
* Add different sets of expected output based on THREAD_MODEL
* Re-add trailing whitespace to `predefined-macros.txt`
@sbc100 wanted to retain the whitespace trailing after certain
predefined macro lines. This change restores that whitespace from
upstream and re-generates the POSIX version using the following command:
Andrew Brown [Tue, 26 Jul 2022 21:15:12 +0000 (14:15 -0700)]
Use MUSL's `weak*` feature in bottom half (#306)
This change extracts the `weak*`-related parts of #303 as a separate PR.
Note that this is slightly strange in that it uses some top-half MUSL
headers in the bottom-half code, but discussion around this led me to
believe that the advantages of, e.g., `LOCK` made this worthwhile.
Beyond just changing uses of `weak` to `__weak__`, we also MUSL's `weak`
and `weak_alias` macros in a few more places.
Dan Gohman [Tue, 24 May 2022 20:56:33 +0000 (13:56 -0700)]
Fix utimensat to avoid passing uninitialized values into WASI calls.
Previously, utimensat would leave the mtim and/or atim timestamps
uninitialized when the `MTIM_NOW` or `ATIM_NOW` were in use, because
that means the respective timestamps are not used.
However, clang now automatically adds `noundef` to the arguments in
functions like `__wasi_path_filestat_set_times`, and there are cases
where simplifycfg can see paths where the uninitialized values are
passed to those `noundef` arguments.
To fix this, change the utimens code to zero out the timestamps when
they aren't in use, to avoid passing uninitialized arguments.
Dan Gohman [Fri, 6 May 2022 01:04:14 +0000 (18:04 -0700)]
Exclude C++ headers from the generated include-all.c program.
The order that things happen in appears to have changed, and the
include-all.c script is now generated at a time when there can be
C++ headers in the sysroot, so adjust the script to exclude C++
headers.
Harald Hoyer [Mon, 2 May 2022 10:21:42 +0000 (12:21 +0200)]
feat: add support for accept and accept4
Since the socket address of the accepted socket is unknown,
all bytes are set to zero and the length is truncated to the size
of the generic `struct sockaddr`.