Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for async ABI, futures, streams, and errors #1895

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

Add support for async ABI, futures, streams, and errors #1895

wants to merge 8 commits into from
+6,774 −337

Conversation

dicej
Copy link
Collaborator

@dicej dicej commented Nov 5, 2024 •
edited
Loading

This adds support for encoding and parsing components which use the Async ABI and associated canonical options and functions, along with the stream, future, and error types.

Note that the error type was recently (about 30 minutes ago) renamed to error-context in Luke's spec PR. I haven't updated this implementation to reflect that yet, but will do so in a follow-up commit. That should allow us to avoid conflicts with existing WIT files that use error as a type and/or interface name. EDIT: done

This does not yet include any new tests; I'll also add those in a follow-up commit.

See bytecodealliance/rfcs#38 for more context.

This is ready for review, but I'll leave it in draft mode until the following are complete:

  • Rename error to error-context
  • Add tests to tests/local/component-model and tests/local/missing-features to exercise new wasm-encoder, wasmparser, and wasmprinter features
  • Add tests to crates/wit-component/tests to exercise its new features
  • Add tests to crates/wit-parser/tests to exercise its new features

@dicej
Add support for async ABI, futures, streams, and errors
015b271 
This adds support for encoding and parsing components which use the [Async
ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md)
and associated canonical options and functions, along with the [`stream`,
`future`, and `error`](WebAssembly/component-model#405)
types.

Note that the `error` type was recently (about 30 minutes ago) renamed to
`error-context` in Luke's spec PR.  I haven't updated this implementation to
reflect that yet, but will do so in a follow-up commit.  That should allow us to
avoid conflicts with existing WIT files that use `error` as a type and/or
interface name.

This does not include any new tests; I'll also add those in a follow-up commit.

See bytecodealliance/rfcs#38 for more context.

Signed-off-by: Joel Dice <[email protected]>
@dicej dicej self-assigned this Nov 5, 2024
@alexcrichton alexcrichton self-requested a review November 5, 2024 23:19
@dicej
Copy link
Collaborator Author

dicej commented Nov 5, 2024

Sorry about all the churn in wasm-compose; I'm currently using it for end-to-end testing, but I could switch to WAC for that and revert most of the wasm-compose changes in this PR if desired.

@dicej
rename error to error-context per latest spec
d5a6c1b 
Signed-off-by: Joel Dice <[email protected]>
@dicej
rename error to error-context per latest spec (part 2)
940de62 
Also, parse string encoding and realloc from encoded `error-context.new` and
`error-context.debug-string` names.

Signed-off-by: Joel Dice <[email protected]>
@alexcrichton
Copy link
Member

Thanks again for this! My current plan is to start tackling this the week-after-next (after WasmCon) but if you've got any high-level points you want me to confirm ahead of time feel free to let me know

@dicej
Copy link
Collaborator Author

dicej commented Nov 6, 2024

Thanks again for this! My current plan is to start tackling this the week-after-next (after WasmCon) but if you've got any high-level points you want me to confirm ahead of time feel free to let me know

No hurry at all; I'm mainly opening this (and soon-to-follow PRs to wit-bindgen and wasmtime) for visibility, i.e. to allow people to track progress if they're interested. The code is still in a pretty raw state (especially the wasmtime parts 😅 ), so it's probably better if you wait a couple of weeks before looking at it, anyway.

This was referenced Nov 6, 2024
Closed
Draft
dicej added a commit to dicej/wasmtime that referenced this pull request Nov 7, 2024
@dicej
Add support for async/streams/futures
05f0998 
This adds support for loading, compiling, linking, and running components which
use the [Async
ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md)
along with the [`stream`, `future`, and
`error-context`](WebAssembly/component-model#405) types.
It also adds support for generating host bindings such that multiple host
functions can be run concurrently with guest tasks -- without monopolizing the
`Store`.

See the [implementation RFC](bytecodealliance/rfcs#38)
for details, as well as [this
repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke
tests.

This is very much a work-in progress, with a number of tasks remaining:

- [ ] Avoid exposing global task IDs to guests and use per-instance IDs instead
- [ ] Track `task.return` type during compilation and assert the actual and expected types match at runtime
- [ ] Ensure all guest pointers are bounds-checked when lifting, lowering, or copying values
- [ ] Reduce code duplication in `wasmtime_cranelift::compiler::component`
- [ ] Add support for `(Typed)Func::call_concurrent` per the RFC
- [ ] Add support for multiplexing stream/future reads/writes and concurrent calls to guest exports per the RFC
- [ ] Refactor, clean up, and unify handling of backpressure, yields, and even polling
- [ ] Guard against reentrance where required (e.g. in certain fused adapter calls)
- [ ] Add integration test cases covering new functionality to tests/all/component_model (starting by porting over the tests in https://github.com/dicej/component-async-demo)
- [ ] Add binding generation test cases to crates/component-macro/tests
- [ ] Add WAST tests to tests/misc_testsuite/component-model
- [ ] Add support and test coverage for callback-less async functions (e.g. goroutines)
- [ ] Switch to back to upstream `wasm-tools` once bytecodealliance/wasm-tools#1895 has been merged and released

Signed-off-by: Joel Dice <[email protected]>
dicej added a commit to dicej/wasmtime that referenced this pull request Nov 7, 2024
@dicej
Add support for async/streams/futures
3bc4a74 
This adds support for loading, compiling, linking, and running components which
use the [Async
ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md)
along with the [`stream`, `future`, and
`error-context`](WebAssembly/component-model#405) types.
It also adds support for generating host bindings such that multiple host
functions can be run concurrently with guest tasks -- without monopolizing the
`Store`.

See the [implementation RFC](bytecodealliance/rfcs#38)
for details, as well as [this
repo](https://github.com/dicej/component-async-demo) containing end-to-end smoke
tests.

This is very much a work-in progress, with a number of tasks remaining:

- [ ] Avoid exposing global task IDs to guests and use per-instance IDs instead
- [ ] Track `task.return` type during compilation and assert the actual and expected types match at runtime
- [ ] Ensure all guest pointers are bounds-checked when lifting, lowering, or copying values
- [ ] Reduce code duplication in `wasmtime_cranelift::compiler::component`
- [ ] Reduce code duplication between `StoreContextMut::on_fiber` and `concurrent::on_fiber`
- [ ] Minimize and/or document the use of unsafe code
- [ ] Add support for `(Typed)Func::call_concurrent` per the RFC
- [ ] Add support for multiplexing stream/future reads/writes and concurrent calls to guest exports per the RFC
- [ ] Refactor, clean up, and unify handling of backpressure, yields, and even polling
- [ ] Guard against reentrance where required (e.g. in certain fused adapter calls)
- [ ] Add integration test cases covering new functionality to tests/all/component_model (starting by porting over the tests in https://github.com/dicej/component-async-demo)
- [ ] Add binding generation test cases to crates/component-macro/tests
- [ ] Add WAST tests to tests/misc_testsuite/component-model
- [ ] Add support and test coverage for callback-less async functions (e.g. goroutines)
- [ ] Switch to back to upstream `wasm-tools` once bytecodealliance/wasm-tools#1895 has been merged and released

Signed-off-by: Joel Dice <[email protected]>
@dicej dicej mentioned this pull request Nov 7, 2024
Draft
17 tasks
@dicej
add wast support for parsing async canon opts
b91ad7d 
And add tests/local/component-model-async/lift-async.wast for round-trip testing
of async lifts (more to come).

Signed-off-by: Joel Dice <[email protected]>
@dicej
more wast async support and more tests
05d06d1 
This also fixes a bug in `wasmprinter` keeping track of core functions.

Signed-off-by: Joel Dice <[email protected]>
@dicej
more wast async support; add async tests; fix bugs
ff6d09f 
Signed-off-by: Joel Dice <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexcrichton alexcrichton Awaiting requested review from alexcrichton

At least 1 approving review is required to merge this pull request.

Assignees

@dicej dicej

Labels
None yet
Projects
Ship WASIp3
Status: In progress
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dicej @alexcrichton