* Make task stats take less memory by default
* Swap order of condition operands
* TaskStats -> ExportedTaskStats
* Add full stats disclaimers
* Clippy
* Fix duplicate labels
* Move stats type reporting to TurboTasks
* Fix turbotrace and turbopack visualization
* Store last duration and execution as SmallDurations
Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
Commit 1: It took me hours to figure out this unsafe problem...
Commit 2: updates tokio
Commit 3: improves hanging detection to allow to attach notes to event listeners
* Sample many modules in benchmarks + reliability fixes
* Fix depth sampling to be uniform
* Fix Webpack benchmark
* Only use detector component for RSC
* Clippy
* Clippy
measure hmr_to_commit with detector component
add Next.js 13 to benchmarks with RSC and RCC measurements
this tests RSC HMR
make HMR warmup faster
make browser launch lazy
test benchmarks for other bundlers on CI too
# Conflicts:
# .github/workflows/test.yml
# crates/next-dev/benches/mod.rs
* use iterations feature from criterion
move startup, warmup and teardown out of the iteration loop to avoid tracking this time as elapsed time
this allows criterion to use the measurement time correctly for the measured task
prefer slope over mean when possible as it estimates the real time (excluding warmup)
* remove warmup argument
* stop server for next iteration
* refactor
* restore verbose info, exclude file read from timing
* add a little bit of delay between HMR updates
* make a warmup change
* no need to copy template dir for HMR benchmark
* add delay after warmup change
* add watch benchmark
* add/fix comments
* fix env var in CI
This PR changes the ecmascript chunk logic to polyfill `global` with `globalThis`. A more complex and less performant solution (but with the benefit of us knowing the runtime environment) would be to add an effect for simple identifier expressions.
* fixup ropes
* add RSC and RCC for turbopack to benchmarks
RSC: whole page is a server component
RCC: whole page is a client component
add `app` directory to test app
* Fix React Refresh boundary finding
We forgot to enable the transform, which means we were rendering from the root of the app instead of the changed component.
* Update snapshots
* Add comment
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Co-authored-by: Anthony Shew <anthony.shew@vercel.com>
Co-authored-by: Jared Palmer <jared@jaredpalmer.com>
Co-authored-by: Maia Teegarden <dev@padmaia.rocks>
Co-authored-by: Will Binns-Smith <wbinnssmith@gmail.com>
Co-authored-by: Alex Kirszenberg <alex.kirszenberg@vercel.com>
Co-authored-by: Matt Pocock <mattpocockvoice@gmail.com>
* Store the sourcemap::SourceMap directly, instead of encoding it
* Implement GenerateSourceMap trait and use it to retrieve live maps
* Cleanup and comments!
* Update safety comment
* Rename var
* Update snapshots
* Update snapshots
* Fix tracing errors in SSR
SSR has access to `EcmascriptChunkVc`, not `EcmascriptChunkContentVc`. Client gets access only to `EcmascriptChunkContentVc`
* Fix source map sources on client
Because the JS is nested in dirs, the source needs to be an absolute path.
Co-authored-by: Jared Palmer <jared@jaredpalmer.com>
* Fix unused question marks warnings
Fix some return types that were giving clippy warnings, mostly `needless_question_mark`.
Remaining warnings are either `too_many_arguments` or `type_complexity`.
* Fix unncessary let binding
Co-authored-by: Justin Ridgewell <justin@ridgewell.name>
Instead of using stdout for communication between the Node.js process
and the Turbopack instance, this PR switches to a TCP connection.
Furthermore, stdout and stderr are now piped between the Node.js process
and Turbopack.
Because the static SSR renderer doesn't emit files that exist in
`node_modules` (because we take advantage of node's native `require`
resolution), any paths that traced into `node_modules` would output the
full file system path. It's just verbose and annoying.
Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
This makes turbopack automatically create `app/layout.js` if
`app/page.js` exists, or `app/layout.tsx` if `app/page.tsx` exists, and
neither layout file already exists.
**Note that I'd prefer this issue to have info severity, but even with a
higher log level of info, the issue is never shown**. Filed vercel/turbo#265 to
track this.
Test Plan:
In a new `create-next-app`, opt into appDirectory and create
`app/page.js`. Run turbopack and verify:
* [x] the index page renders correctly
* [x] An issue with Warning level is logged notifying the user an
`app/layout.js` was created
* [x] `app/layout.js` is actually created with basic root layout
In a new `create-next-app`, opt into appDirectory and create
`app/page.tsx`. Run turbopack and verify:
* [x] the index page renders correctly
* [x] An issue with Warning level is logged notifying the user an
`app/layout.tsx` was created
* [x] `app/layout.tsx` is actually created with basic root layout
In a new `create-next-app`, opt into appDirectory and create
`app/page.tsx`. Also create a basic `app/layout.js` (note js layout with
tsx page). Add a custom `<title>` to the layout. Run turbopack and
verify:
* [x] the index page renders correctly
* [x] No issue is logged
* [x] No files are modified
* [x] The custom `<title>` is reflected.
This adds the Next SSG transform. This transform does a few things, but
the one we're most interested in right now is the removal of
`getStaticProps`/`getServerSideProps`/etc. from page modules. HMR will
be disabled if these exports are preserved.
Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
Seems like app layout was left behind while shipping features to normal
SSR rendering
* fixes fallback error pages (update to error fallback changes)
* fixes process communication (update to api route changes)
* avoid busy looping
* update to next.js updates
* fix resolving cycle
* fix HMR of client components
* add some hard coded external packages
Enabled in next-dev and snapshot tests.
There are quite a few output snapshot files. Maybe we make stubs for the
third-party packages to reduce the noise...
Test Plan: Added new snapshot tests. Verified transformed code contains
a digest, inline source map string.
This reworks our Node.js rendering logic to allow for:
* passing binary request bodies in and out of Node.js: API routes can
accept request bodies in POST, and can reply with any content type.
* content source results that are recomputed every time: we don't want
API routes to be cached (at least not by turbopack for now).
It also reworks the `END_OF_OPERATION` logic to avoid hard coding a
single end of operation marker (they're now unique per pool), and allow
multiple kinds of operation events (`Step`, `Success`, `Error`).
This is not a particularly good implementation for this. A better
implementation would proxy the request from the client to the Node.js
server in a more direct manner. We also need a way to tell turbo tasks
"don't bother ever caching this", as right now every single API request
and result will still be cached, even if we know they will never be used
again. Finally, we should communicate with Node.js processes via a
better mechanism than stdout. I made some progress on a prototype for
using https://github.com/servo/ipc-channel with NAPI a while back, which
I'd like to pick up some time after conf.
However, it seems to work well enough for now, so yay.
Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
Closesvercel/turbo#109
Note a difference between this and stable Next.js: at turbopack this
file is placed in .next/server/package.json, while Next.js currently
creates this file at .next/package.json.
Test Plan: Verified that the `hello-world-esm` no longer fails to SSR
(it's now blocked by vercel/turbo#111).
Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
Small nit, but if we want to listen on 0.0.0.0, then we should set that
to the default. The difference is that other computers can connect when
we listen on 0.0.0.0, while only this computer can connect if we listen
on 127.0.0.1.
Note that this doesn't completely close out
[vercel/turbo#54](https://github.com/vercel/web-tooling-internal/issues/54) as the
[tailwindcss config is
optional](https://tailwindcss.com/docs/configuration). I'm digging
deeper at additional ways to detect tailwind, but putting this up since
it's a strict improvement.
```
olszewski@chriss-mbp next-dev % cargo run -p next-dev -- /tmp/with-tailwindcss-app
Compiling next-dev v0.1.0 (/Users/olszewski/code/vercel/the-three-body/crates/next-dev)
Finished dev [unoptimized + debuginfo] target(s) in 11.61s
Running `/Users/olszewski/code/vercel/the-three-body/target/debug/next-dev /tmp/with-tailwindcss-app`
server listening on: http://localhost:3000
error [unimplemented]
/private/tmp/with-tailwindcss-app/next.config.js
Feature not yet supported
Handling the file `next.config.js` is currently unimplemented
/private/tmp/with-tailwindcss-app/postcss.config.js
Feature not yet supported
Handling the file `postcss.config.js` is currently unimplemented
/private/tmp/with-tailwindcss-app/tailwind.config.js
Feature not yet supported
Handling the file `tailwind.config.js` is currently unimplemented
[200] / (3045ms)
initial compilation 3468ms (3044ms task execution, 48380 tasks)
updated in 874ms (8990 tasks)
```
Ported from https://github.com/vercel/turbo-tooling/pull/452.
To do:
* [x] Read and use node version from PATH
Test Plan:
Temporarily lowered the default version and verified core-js was
resolved when /page is visited.
Co-authored-by: Justin Ridgewell <justin@ridgewell.name>
Fixes a few issues:
1. Removes `Hasher` impl for `Xxh3Hash64Hasher`
- Importing the `Hash` trait allows you to call `vc.hash(&mut
xxh3_hasher)`, which defeats our goal of deterministic hashing
2. Fixes hashing of 2 contiguous strings
- `"fo"` and `"o"` should hash differently than `"f"` and `"oo"`.
3. Fixes hashes of enums
- `Foo::Bar(1)` should hash differently than `Foo::Baz(1)`
4. Standardizes `usize`/`isize` as 8 bytes
Since we used to build the HTML using our own `<Document>` component, we
were previously adding a data-turbopack-chunk-id attribute to our
`<link>` tags to reconcile chunk paths with their chunk ids when
initializing HMR. However, Next.js is now responsible for building the
HTML, and it has no such mechanism.
**NOTE:** HMR is currently broken for non-Next-SSR rendering
(HtmlAsset). This PR does not fix that.
* remove 100 modules form the default modules count
* rename bench_restart -> bench_hydration_cached
* add bench_startup_cached
* run npm install only once per test case and copy the result instead
* use multi-threaded copy
* Run `cached` tests only when `TURBOPACK_BENCH_CACHED` is set
* add `TURBOPACK_BENCH_BENCH` too measure the time it takes to run the
full benchmark (including startup and teardown) instead.
* add `TURBOPACK_BENCH_HEAD` to run non-headless
* add `TURBOPACK_BENCH_DEVTOOLS` to auto open devtools
* retry initial warmup change when it fails to detect that
This should make the benchmarks about 3 times faster... at least on
windows^^
Rebase of vercel/turbo#68, which was merged into a now-closed PR.
Closesvercel/turbo#65.
When configured to use preset-env, this stops turbopack from inserting
dependencies on core-js. It's:
* Buggy/error-prone, at least through swc preset-env:
* https://github.com/vercel/the-three-body/issues/65
* setImmediate "polyfills" always inserted for this nonstandard feature
when
React (scheduler, iirc) does feature detection of it
* Big. Updating the snapshot alone for a test that only used [].includes
resulted in a diff with nearly 4700 line deletions
* Not done by Next.js for modern targets in the past couple of years:
https://twitter.com/timneutkens/status/1234548900272517120
Test Plan: Added [].includes to preset_env snapshot test and verified no
references to core-js are inserted.
This will remove the warning about using `hydrate` instead of
`hydrateRoot`, and fix a mismatch between SSR and CSR when using
styled-jsx (`<style jsx>`).
We can't use the normal `ImportMap` because we stop resolving if looking
up a request within yields a result. However, for module polyfills, we
only need to look up the request when normal resolving fails. Hence the
dichotomy between the `import_map` and the `fallback_import_map`.
In a more modular architecture, we would model this as a stack of three
resolvers:
1. ImportMapResolver
2. DefaultResolver
3. FallbackImportMapResolver
I'm guessing this is what we'll have in the future, but this requires a
bigger refactoring than what I'm comfortable with implementing for now.
This PR also includes the changes in vercel/turbo#12
This optimizes Next.js SSR by marking `react[/*]` and `next[/*]` imports
as externals in Node.js, which means that they will be `require`d
directly instead of bundled.
This optimization reduces the number of modules to analyze on the server
for a single component app from ~1000 (Next.js SSR imports a lot of
modules, including amp-optimizer which includes more) to <10.
Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
add very simple explorer to introspect the graph via `/__turbopack__/`
Allows to "walk" through the content sources and asset graph. Some
assets allows to access inner assets, e. g. from the ecmascript chunk
you can access the entry module asset, so you can go a level deeper.
It works by a `Introspectable` trait which can be implemented by
anything we want to allow the user to inspect. In future the in browser
UI should offer a nice UI to explore your application.
In future we probably also want to add some `metrics` to the
`Introspectable` trait, e. g. to report size of an asset.
This UI is very basic currently, but that works for debugging:
It's important to not create a `cell` outside of the root `source` method as this will create a new cell for each call, which means each `source` will create a separate task, and this bubbles for each cell created in `source`...
This makes the argument to `embed_file!` relative to the package it is invoked from, instead of relative to the file it is invoked from. The latter is closer to how `include_*!` works, but it has a couple of issues:
* Since `file!` returns a path relative to the workspace, it requires knowing the full path to the workspace. However, this is undefined when the package is compiled from outside of the workspace (e.g. as a dependency of another package).
* Procedural macros don't have access to the path of the file where they were originally invoked [(although we could get it with #[feature(proc_macro_span)])](https://doc.rust-lang.org/proc_macro/struct.Span.html), which means that an `embed_dir` macro would not be able to be relative to the invoker file anyway. So this makes things more consistent for such a (to be implemented) macro.
This is a very simplified version of what Next.js does. It adds support for `App` and page components defining `getInitialProps`.
fixesvercel/web-tooling-internal#38
This updates all of our Issues to have a consistent display, and changes the way we display source code and debug-level details. In particular, this makes the following changes:
1. Cleans up the title/description of existing issues
- We were filling the `title` field with too much information, because the `desccription` field wasn't always shown.
1. Always displays the `description`
- This used to be hidden behind the `--detail` flag
1. Always displays the code frame of the issue, if it exists
- This used to be hidden behind the `--detail` flag
1. Fixes the `context` (file path) to display the relative path to the file
- It was hardcoded to the FS string used by turbotrace
1. Creates a new `detail` function which can provide debug-level information about the internal Rust state
- This is not shown by default, user must pass `--detail` flag
The format for a rendered issue roughly matches:
```
${SEVERITY} [${CATEGORY}]
${CONTEXT}
${TITLE}
${CODE FRAME}
${DESCRIPTION}
${DETAIL}
```
Where
- `SEVERITY` is error/warning/etc
- `CATEGORY` is a self-defined string for the phase of the compiler (eg, "parsing", "running")
- `CONTEXT` is a file path to the offending file
- `TITLE` is a short (single-line) title
- `CODE_FRAME` is an optional position that generated the issue, which will display a few context lines and highlight the offending span of code
- `DESCRIPTION` is a longer (multi-line) message that describes the issue, and what to do about it
- `DETAIL` is a (by default not shown) fully detailed, debug-level, message providing insights into the internal state of Rust
This adds preliminary support for pages/_document. The API is more complex than pages/_app, so this required a few more things in order to work.
Right now, our `next/document` shim does much, much less than the original. We will eventually have to figure out whether we should distribute our own module, or find a way to reuse Next.js' internals. There's *a lot* of stuff in there so the second option is probably best.
This adds preliminary support for pages/_app. There's still some work to do here to support things like `App. getInitialProps`, etc., but that will come in later PRs. I've created issues for these for now.
One issue remaining is that renaming `pages/_app.js` does not switch to the default _app asset. The server needs to be restarted for this to happen. Similarly, adding an _app.js file does not switch from the default asset to it. I'm guessing there's something wrong with the way I set up the import mappings, and they aren't being invalidated properly?
fixesvercel/turbo#424
remove context_path from AssetContext
remove most with_* methods from AssetContext
add ResolveOrigin trait
use ResolveOrigin to pass along context path and context for resolving
ecmascript and css module assets implement ResolveOrigin
Noteable change: Many places use `origin_path` instead of `context_path` which means it points to the issuer module path instead of the issuer module directory.
Closesvercel/turbo#440
This uses swc's preset_env to automatically downlevel code according to the environment's browser targets, and sets next-dev to use a limited, modern target.
To do:
* [x] Add snapshot test
* [x] turbotrace test failures — looks like this has something to do with the Buffer module? Hitting swc's "multiple constructors" not implemented: f655488cfa/crates/swc_ecma_transforms_compat/src/es2015/classes/mod.rs (L545)
* [x] ~Benchmark downleveling node_modules (probably on front) and make a decision re: downleveling everything vs. just workspaces~ Filed as vercel/turbo#457
Wrapper is a bit of a weird name since it's not actually wrapping the existing asset. Actually it's just a virtual asset with a path below the existing path.
* Fix DiskFileSystem::read_link
* pnpm-like integration test
* Introduce AssetContent to handle symlink assets
* Fix read_link on Windows
* Run clippy fix
* Rename `path` to `target`
Co-authored-by: Justin Ridgewell <justin@ridgewell.name>
* Split Windows specified code
* Add comments about Redirect is only represent Directory
* Handle symlink while reading content
* Clippy fix
* Revert previous changes in FileSystemPathVc::get_type
* Fix Unix read_link
* cleanup
* handle symlink while resolving native bindings
* Make LinkContent::Link contains only target
* Add LinkType to represent link type
* Cleanup VersionedAsset
* Cleanup LinkType
* Normalize the LinkContent::target
* Comments
* Revert special case workaround for sharp on Windows
* comments
* node_native_binding follow file link
* Apply CR suggestion
Co-authored-by: Justin Ridgewell <justin@ridgewell.name>
This implements `.env` loading by taking advantage of the [dotenvy](https://docs.rs/dotenvy/latest/dotenvy/) crate. Unfortunately, this isn't as featureful as the npm `dotenv`, lacking `FOO=${BAR:-default}` default support (this might be important), `FOO=${MAYBE_UNDEFINED:-${BAR-:default}}` nested fallbacks, and `` FOO=`backtick` `` support.
This is then converted into a series of assignments for the client code. This is run before any of the user code.
```js
const env = process.env;
env["FOO"] = "bar";
//...
```
- - -
TODO:
- [x] ~~Replace `process.env.FOO`/`process.env["FOO"]` during chunk compilation~~
- using a runtime module speeds up HMR because the code doesn't need to be recompiled.
- [x] I don't actually know how to test server rendering, so I assume it works but haven't verified.
