This PR also includes the changes in vercel/turbo#12
This optimizes Next.js SSR by marking `react[/*]` and `next[/*]` imports
as externals in Node.js, which means that they will be `require`d
directly instead of bundled.
This optimization reduces the number of modules to analyze on the server
for a single component app from ~1000 (Next.js SSR imports a lot of
modules, including amp-optimizer which includes more) to <10.
Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
add very simple explorer to introspect the graph via `/__turbopack__/`
Allows to "walk" through the content sources and asset graph. Some
assets allows to access inner assets, e. g. from the ecmascript chunk
you can access the entry module asset, so you can go a level deeper.
It works by a `Introspectable` trait which can be implemented by
anything we want to allow the user to inspect. In future the in browser
UI should offer a nice UI to explore your application.
In future we probably also want to add some `metrics` to the
`Introspectable` trait, e. g. to report size of an asset.
This UI is very basic currently, but that works for debugging:
It's important to not create a `cell` outside of the root `source` method as this will create a new cell for each call, which means each `source` will create a separate task, and this bubbles for each cell created in `source`...
This makes the argument to `embed_file!` relative to the package it is invoked from, instead of relative to the file it is invoked from. The latter is closer to how `include_*!` works, but it has a couple of issues:
* Since `file!` returns a path relative to the workspace, it requires knowing the full path to the workspace. However, this is undefined when the package is compiled from outside of the workspace (e.g. as a dependency of another package).
* Procedural macros don't have access to the path of the file where they were originally invoked [(although we could get it with #[feature(proc_macro_span)])](https://doc.rust-lang.org/proc_macro/struct.Span.html), which means that an `embed_dir` macro would not be able to be relative to the invoker file anyway. So this makes things more consistent for such a (to be implemented) macro.
This is a very simplified version of what Next.js does. It adds support for `App` and page components defining `getInitialProps`.
fixesvercel/web-tooling-internal#38
This updates all of our Issues to have a consistent display, and changes the way we display source code and debug-level details. In particular, this makes the following changes:
1. Cleans up the title/description of existing issues
- We were filling the `title` field with too much information, because the `desccription` field wasn't always shown.
1. Always displays the `description`
- This used to be hidden behind the `--detail` flag
1. Always displays the code frame of the issue, if it exists
- This used to be hidden behind the `--detail` flag
1. Fixes the `context` (file path) to display the relative path to the file
- It was hardcoded to the FS string used by turbotrace
1. Creates a new `detail` function which can provide debug-level information about the internal Rust state
- This is not shown by default, user must pass `--detail` flag
The format for a rendered issue roughly matches:
```
${SEVERITY} [${CATEGORY}]
${CONTEXT}
${TITLE}
${CODE FRAME}
${DESCRIPTION}
${DETAIL}
```
Where
- `SEVERITY` is error/warning/etc
- `CATEGORY` is a self-defined string for the phase of the compiler (eg, "parsing", "running")
- `CONTEXT` is a file path to the offending file
- `TITLE` is a short (single-line) title
- `CODE_FRAME` is an optional position that generated the issue, which will display a few context lines and highlight the offending span of code
- `DESCRIPTION` is a longer (multi-line) message that describes the issue, and what to do about it
- `DETAIL` is a (by default not shown) fully detailed, debug-level, message providing insights into the internal state of Rust
This adds preliminary support for pages/_document. The API is more complex than pages/_app, so this required a few more things in order to work.
Right now, our `next/document` shim does much, much less than the original. We will eventually have to figure out whether we should distribute our own module, or find a way to reuse Next.js' internals. There's *a lot* of stuff in there so the second option is probably best.
This adds preliminary support for pages/_app. There's still some work to do here to support things like `App. getInitialProps`, etc., but that will come in later PRs. I've created issues for these for now.
One issue remaining is that renaming `pages/_app.js` does not switch to the default _app asset. The server needs to be restarted for this to happen. Similarly, adding an _app.js file does not switch from the default asset to it. I'm guessing there's something wrong with the way I set up the import mappings, and they aren't being invalidated properly?
fixesvercel/turbo#424
remove context_path from AssetContext
remove most with_* methods from AssetContext
add ResolveOrigin trait
use ResolveOrigin to pass along context path and context for resolving
ecmascript and css module assets implement ResolveOrigin
Noteable change: Many places use `origin_path` instead of `context_path` which means it points to the issuer module path instead of the issuer module directory.
Closesvercel/turbo#440
This uses swc's preset_env to automatically downlevel code according to the environment's browser targets, and sets next-dev to use a limited, modern target.
To do:
* [x] Add snapshot test
* [x] turbotrace test failures — looks like this has something to do with the Buffer module? Hitting swc's "multiple constructors" not implemented: f655488cfa/crates/swc_ecma_transforms_compat/src/es2015/classes/mod.rs (L545)
* [x] ~Benchmark downleveling node_modules (probably on front) and make a decision re: downleveling everything vs. just workspaces~ Filed as vercel/turbo#457
Wrapper is a bit of a weird name since it's not actually wrapping the existing asset. Actually it's just a virtual asset with a path below the existing path.
* Fix DiskFileSystem::read_link
* pnpm-like integration test
* Introduce AssetContent to handle symlink assets
* Fix read_link on Windows
* Run clippy fix
* Rename `path` to `target`
Co-authored-by: Justin Ridgewell <justin@ridgewell.name>
* Split Windows specified code
* Add comments about Redirect is only represent Directory
* Handle symlink while reading content
* Clippy fix
* Revert previous changes in FileSystemPathVc::get_type
* Fix Unix read_link
* cleanup
* handle symlink while resolving native bindings
* Make LinkContent::Link contains only target
* Add LinkType to represent link type
* Cleanup VersionedAsset
* Cleanup LinkType
* Normalize the LinkContent::target
* Comments
* Revert special case workaround for sharp on Windows
* comments
* node_native_binding follow file link
* Apply CR suggestion
Co-authored-by: Justin Ridgewell <justin@ridgewell.name>
This implements `.env` loading by taking advantage of the [dotenvy](https://docs.rs/dotenvy/latest/dotenvy/) crate. Unfortunately, this isn't as featureful as the npm `dotenv`, lacking `FOO=${BAR:-default}` default support (this might be important), `FOO=${MAYBE_UNDEFINED:-${BAR-:default}}` nested fallbacks, and `` FOO=`backtick` `` support.
This is then converted into a series of assignments for the client code. This is run before any of the user code.
```js
const env = process.env;
env["FOO"] = "bar";
//...
```
- - -
TODO:
- [x] ~~Replace `process.env.FOO`/`process.env["FOO"]` during chunk compilation~~
- using a runtime module speeds up HMR because the code doesn't need to be recompiled.
- [x] I don't actually know how to test server rendering, so I assume it works but haven't verified.
Many small things I found
The most important is probably the typescript transform
The remaining bits should hopefully be self-explanatory from the commit messages
With React.memo:
```
bench_hmr_to_commit/Turbopack CSR/30000 modules
time: [50.608 ms 51.659 ms 52.553 ms]
```
Without React.memo:
```
bench_hmr_to_commit/Turbopack CSR/30000 modules
time: [853.47 ms 1.0191 s 1.1873 s]
change: [+1543.4% +1872.7% +2207.8%] (p = 0.00 < 0.05)
Performance has regressed.
```
Since we're only ever editing the top-level triangle in our HMR benchmarks, we're incurring the time it takes for React to re-render the whole tree, which is a function of the number of components in said tree. By using `React.memo`, we can skip updating children components during HMR.
moves the logic of creating SourceMap assets into the asset reference. This avoids depending on the chunk items in the references() method directly. It also avoids calling CodeVc::source_map() until the source map is read
avoid circular dependency in call graph
It also avoids checking `has_source_map()` and just inserts potential Source Maps assets for every chunk item. Checking `has_source_map()` seems unnecessary work to do for all chunk items, when we can just send an empty source map.
only expose SourceMaps for chunk items when HMR is enabled
picked typescript transform from https://github.com/vercel/turbo-tooling/pull/341
add resolve options context as global resolve config
enable typescript only for next-dev
move emulating logic from environment to resolve options context
Co-authored-by: Leah <8845940+ForsakenHarmony@users.noreply.github.com>
This implements support for styled-jsx in next-dev using swc's styled_jsx crate.
It's only applied in next-dev, and is only applied as a transform to app code, much like the react-refresh transform.
To do:
* [x] The transform doesn't seem to be applied. Pass the added test.
Test Plan: `cargo test -p next-dev --
test_crates_next_dev_tests_integration_turbopack_basic_styled_jsx
--nocapture`
Remaining questions:
* Should we have some static analysis for `getStaticProps` instead of looking into exports at runtime?
* For now, the output of `getStaticProps` (if defined) will always trump the value passed in as `data`. If we consider `data` to be the cached output of `getStaticProps` (in the future, as this is not yet implemented), this logic should be adapted.
Previously, we ran multiple `npm install` operations in serial using multiple calls to `install_from_npm`. Instead, this allows us to express dependencies all at once as a single command to the npm cli, which should reduce the time we spend installing from npm and updating package.json.
Test Plan: Manually confirmed that package.json was updated correctly. `cargo bench`.
This adds webpack 5 to the benchmark suite.
Test Plan: Manually confirmed package.json updates and webpack config written to temp dir correctly. `cargo bench`.
This splits the benchmark code into more modules. Notes:
* ~Moved/left `get_bundlers()` and `get_module_counts()` to/in mod.rs. In particular, moving `get_bundlers()` to either bundle.rs or util.rs would lead to a circular dependency. These both also rely on env var configuration, so I figured this was a reasonable place for them.~
* The Bundler trait has its own module (not moved to util), since it's a top-level concern and not really a miscellaneous utility.
* Each bundler has its own module file.
Test Plan: `TURBOPACK_BENCH_BUNDLERS=all cargo test --benches -p next-dev -- --nocapture` and verify same output as before change.
* Benchmark Parcel
* add Parcel to the CI benchmarks
* move some turbopack dependencies to the bundler as they conflict with other bundlers
Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
This implements benchmark support for Next.js 12. Next.js (the tool) expects to be able to resolve from the `next` package in the cwd, so it must be installed alongside the other node_modules in the test. `prepare` was added to the Bundler trait to handle this case.
Test Plan: `TURBOPACK_BENCH_ALL=all cargo bench -p next-dev`
Co-authored-by: Alex Kirszenberg <1621758+alexkirsz@users.noreply.github.com>
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
This adds a comparison against Vite to our benchmark suite, running the startup, change, and restart benchmarks.
Test Plan: `cargo bench`
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
* avoid cloning Strings and reading/calling many functions when building ecmascript chunk content
* introduce `ReadRef` to allow storing snapshot of a value in values
* use snapshot trees to allow caching more function calls and reads
Before:
```
updated in 35ms (18 tasks)
updated in 34ms (18 tasks)
updated in 37ms (18 tasks)
updated in 31ms (18 tasks)
updated in 40ms (19 tasks)
updated in 37ms (18 tasks)
updated in 37ms (18 tasks)
updated in 34ms (18 tasks)
updated in 35ms (18 tasks)
updated in 52ms (18 tasks)
```
After:
```
updated in 6.105ms (19 tasks)
updated in 5.279ms (18 tasks)
updated in 10.471ms (19 tasks)
updated in 6.863ms (18 tasks)
updated in 4.593ms (18 tasks)
updated in 4.173ms (18 tasks)
updated in 5.352ms (18 tasks)
updated in 10.69ms (18 tasks)
updated in 5.065ms (18 tasks)
updated in 6.309ms (19 tasks)
```
a 5x performance improvement
This implements the basics of parameterizing the tool/devserver used in these tests. Following PRs will implement benchmarking of Vite, bun, Parcel, etc.
Test Plan: `cargo bench -p next-dev` and verify no change in performance.
This implements a benchmark of restarting the devserver after successfully starting it and shutting it down.
## Question/TODO:
Since our goal is metrics that don't scale with project size, should we
assert that the small/medium benchmark results don't differ?
Test Plan: `cargo bench -p next-dev`
This builds on vercel/turbo#240, starting up a server and then benchmarking the response to a small file change.
This change does not introduce nor remove any dependencies. A followup
benchmark should do so.
Test Plan: cargo bench -p next-dev
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Instead of dropping back to prefix calling form in `try_join_all(iterator_of_intofutures).await?`, we now use postfix form `iterator_of_intofutures.try_join().await?`.
This adds an assertion that no runtime (browser) errors occurred when
loading a benchmark page.
Test Plan: Temporarily removed the npm install for the test app and
verified the benchmark failed as the test app requires react, react-dom.
Restored the npm install and verified the benchmark runs to completion.
This:
* Runs `npm install` in test directories to provide turbopack with modules necessary to bundle them.
* Reuses test directories for iterations across the given benchmark. This prevents unnecessary file writing and `npm install` for each iteration, improving the times to run benchmarks.
Currently cherry-picks vercel/turbo#278 as it's necessary along with vercel/turbo#277.
Test Plan: Connected to the running devserver mid-test and confirmed no errors are thrown and the triangle is rendered correctly.
This PR implements HMR support with React Refresh built-in.
For now, in order for React Refresh to be enabled, you'll need the `@next/react-refresh-utils` package to be resolveable: `yarn add @next/react-refresh-utils` in your app folder.
* Depends on vercel/turbo#266
* Integrated both HMR-and-React-Refresh-specific logic directly into the ES chunks' runtime. Webpack has more complex setup here, but for now this makes the logic much more easy to follow since everything is in one place. I have yet to implement the "dependencies" signature for `hot.accept`/`hot.dispose`, since React Refresh does not depend on them. We'll have to see if they're even used in the wild or if we should deprecate them.
* Only implemented the [module API](https://webpack.js.org/api/hot-module-replacement/#module-api), not the [management API](https://webpack.js.org/api/hot-module-replacement/#management-api). We apply all updates as soon as we receive them.
* Added support for "runtime entries" to ES chunks. These are assets that will be executed *before* the main entry of an ES chunk. They'll be useful for polyfills in the future, but for now they're here to evaluate the react refresh runtime before any module is instantiated.
Next steps for HMR:
* Implement CSS HMR
* Implement (or decide to deprecate) the [dependencies form](https://webpack.js.org/api/hot-module-replacement/#accept) of `hot.accept`/`hot.dispose`
* Clean up `runtime.js` some more: switch to TypeScript, split into multiple files, etc. It'd be nice if all of this could be done at compile time, but how to achieve this is unclear at the moment. _Can we run turbopack to compile turbopack?_
* Basic startup bench for dev server
* fixes to benchmarking (vercel/turbo#268)
* use bench profile for benchmarking
* make setup and teardown not part of the measurement
add support for async setup and teardown
share browser between measurements
* updates for changes TestApp
Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
The snapshot tests were failing because Windows paths were sneaking into the `FileSystemPathVc::path`. Reviewing, the `::new` method didn't normalize `\` into `/`, and the various `::join` methods didn't either. This came up in both the chunk ids, and the request pathnames used by the server.
No path with backslash should enter the FileSystemPath APIs, they should be normalized during conversion from `Path` to `String`
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
There are a bunch or problems with invalidations:
* The fs impl watches path case-insenstive. This means two paths might conflict when on a case-sensitive filesystem. It uses an array of Invalidators now
* Move the next-dev bootstrapping logic out of the run_once scope (which is not updated when invalidations occur). Instead it's executed inside the request handling resp. update stream where changes can be handled.
* `TransientValue` was not an Value type actually. This fixes that.
* Adds a new `TransientInstance` wrapper to pass transient by reference.
* `strongly_consistent` was broken when using nested TaskScopes. This fixes that.
Server Rendering:
* This adds an additional ContentSource to next-dev which takes care of handling the `pages` directory.
* The content source creates a ServerRenderedAsset from each file in the `src/pages` or `pages` directory and a AssetGraphContentSource for that.
* The ServerRenderedAsset will reference an underlying asset for the node.js context which will be passed to the node executable for rendering. It uses a WrapperAsset to add additional communication logic.
Client Transition:
* When annotating `import`s with `transition: "next-client"` the NextClientTransition is used
* This transition changes the environment to browser
* It wraps the referenced asset with a next-hyrdation wrapper asset
* It leaves a little module in the previous context which exports a list of URLs for chunks needed.
* The NextClientTransition takes a client_chunking_context as argument which specifies how the client code is chunked.
These integration tests have been flaky, failing when a "free" port turns out to be in use. Since nextest parallelizes test runs and portpicker guesses and checks free ports [0], I'm guessing that there's a collision occurring.
Instead, ask the operating system for a free port by binding to port 0 and read the port back from the resulting address.
Test Plan: Tried local runs with nextest, but those succeeded before as well. I'll probably retry things on CI a few times.
[0] 912f913ac3/src/lib.rs (L53)
This adds test runs for integration tests in `__skipped__` directories,
ensuring that they fail, otherwise they should probably be unskipped.
Test Plan: Temporarily moved a succeeding test into a `__skipped__`
directory and ensured that cargo test began failing that test.
This commits webpack's chunk tests (test/cases/chunks) and skips those
that do not pass yet.
Test Plan: `cargo test -p next-dev -- --nocapture` and verify the
non-skipped tests run.
This prepares the way for HMR (vercel/turbo#160) by letting us diff assets between
versions.
1. Add `Asset::versioned_content` which returns a `VersionedContentVc`.
2. `VersionedContent`s have a built-in versioning mechanism as they must implement `version() -> VersionVc`. `Version` is a trait, so `VersionVc` can contain a specific version implementation by asset type. This is particularly important because...
3. A `VersionedContentVc` can be diffed with a `VersionVc` from the same underlying `VersionedContent` type with `content.update(from: version)`. This returns an `UpdateVc` which describes the steps necessary to update from one verson to the next. In the case of ES chunks, this will be a map of added, and modified module IDs, with their respective factories, and a set of deleted module IDs.
4. Implement diffing for ES chunks.
This implements a manual debug mode for next-dev tests, enabled by
setting the environment variable TURBOPACK_DEBUG_BROWSER to any value.
It launches the test browser in non-headless mode and holds it open
~~indefinitely~~ until the user closes it, so it can be inspected.
Test Plan: `TURBOPACK_DEBUG_BROWSER=1 cargo test -p next-dev --
test_crates_next_dev_tests_integration_chunks_circular_correctness
--nocapture` and verify the browser is opened non-headless and is held
open
This is a very early version of the next-dev test runner. I'm opening this early to get thoughts from folks re: the direction of the design and implementation.
Fixesvercel/turbo#204
Currently it:
* Discovers integration test fixtures from the filesystem. Right now these are expected to be single files that get bundled and will eventually include assertions. This is powered by the test-generator crate, which allows us not to have to manually enumerate each case. We could consider using this for the node-file-trace tests as well.
* Starts the dev server on a free port and opens a headless browser to its root. The browser control is implemented with the https://crates.io/crates/chromiumoxide crate, which expects Chrome or Chromium to already be available.
Eventually it will:
* [x] Implement a minimal test environment loaded in the browser so that assertions can be run there from bundled code.
* [x] Report back the results of these assertions to rust, where we can pass/fail cargo tests with those results.
In the future it could:
* Possibly include snapshot-style tests to assert on transformed results. This could be in the form of fixture directories instead of files cc @jridgewell
* Support expressing special configuration of turbopack in a fixture, possibly as another file in the fixture directory.
* [x] ~Possibly support distributing tests to a pool of open browsers instead of opening and closing for each test.~
Test Plan: See next PRs
This PR builds our own `Debug`-like derive-macro machinery for formatting structs, relying on `std::fmt::Formatter` for the actual formatting.
### Usage
A new `ValueDebug` trait is automatically implemented for all `#[turbo_tasks::value]`s, which has a single `.dbg()` method which resolves to a debug representation that can then be printed to the screen. `#[turbo_tasks::value_trait]` also implement the `.dbg()` method directly.
```rust
dbg!(any_vc.dbg().await?);
```
If you have a `#[turbo_tasks::value]` struct with a field that doesn't implement `Debug`, you'll want to declare that field as `#[debug_ignore]`. For instance:
```rust
#[turbo_tasks::value(ContentSource, serialization: none, eq: manual, cell: new, into: new)]
pub struct TurboTasksSource {
#[debug_ignore]
#[trace_ignore]
pub turbo_tasks: Arc<TurboTasks<MemoryBackend>>,
}
```
### Why not use `Debug` directly?
We can't use `Debug` because our values are resolved asynchronously and can nest `Vc`s arbitrarily. I tried using `futures::executor::block_on` to resolve them synchronously in a `Debug` implementation but that causes deadlocks.
Cherry-picking this from my work on the next-dev test runner.
This moves browser opening from the turbopack-dev-server crate into the next-dev crate, which has the cli entrypoint that runs the dev server. It looks like the dev server package is meant to be used as a library (it's only a library crate), and having this external side effect feels unexpected and makes it difficult to use this crate in situations like a test runner for next-dev, where we should test with a headless web browser.
Alternatively, opening the browser could be an option passed when creating the dev server, but this feels a bit cleaner to me.
Test Plan: `cargo run -p next-dev` and verify the browser still opens and successfully connects to the dev server.
* Initialize Node.js/TypeScript workspace
* node-module-trace Webpack plugin
* Add new fmt checks to pipeline
* Popup unwind error
* Implement --exact flag
* Yarn 3.2.2
* Reformat toml files
* Fix socket io test, 100ms timeout is too long
* remove unnecessary CI cache config
* regenerate lockfile from old lockfile, align the dependencies version
* Run nmt tests in system tmp dir
* Apply code review suggestions
* allow to wait for task completion and propagate errors and panics
* revert method addition
* spawn_root_task should be sync
Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
make TransientValue functional
make console-subscriber an optional feature
avoid cloning when calling a function
capture timings of task executions
fix LazyAsset
expose graph from dev server
