There should be no shared react packages in our server runtime. rsc should always be separate from ssr.
This update reconfigures the runtiem to eliminate shared react modules. the jsx runtime will now be separate for RSC and SSR. this is necessary because the implementations for the jsx runtime rely on React and they need to see the right versions.
Additionally I fixed an alias so that the shared subset react is used when using react-server condition.
I also fixed a bug in 2 tests related to class/className.
Note: this PR blocks upgrading React canary due to internal changes in how React state is managed in when using the `react-server` condition
Today when we hydrate an SSR'd RSC response on the client we encounter import chunks which initiate code loading for client components. However we only start fetching these chunks after hydration has begun which is necessarily after the initial chunks for the entrypoint have loaded.
React has upstream changes that need to land which will preinitialize the rendered chunks for all client components used during the SSR pass. This will cause a `<script async="" src... />` tag to be emitted in the head for each chunk we need to load during hydration which allows the browser to start fetching these resources even before the entrypoint has started to execute.
Additionally the implementation for webpack and turbopack is different enough that there will be a new `react-server-dom-turbopack` package in the React repo which should be used when using Turbopack with Next.
This PR also removes a number of patches to React src that proxy loading (`__next_chunk_load__`) and bundler requires (`__next_require__`) through the `globalThis` object. Now the react packages can be fully responsible for implementing chunk loading and all Next needs to do is supply the necessary information such as chunk prefix and crossOrigin attributes necessary for this loading. This information is produced as part of the client-manifest by either a Webpack plugin or Turbopack.
Additionally any modifications to the chunk filename that were previously done at runtime need to be made in the manifest itself now. This means we need to encode the deployment id for skew protection and encode the filename to make it match our static path matching (and resolutions on s3) when using `[` and `]` segment characters.
There are a few followup items to consider in later PRs
1. we currently bundle a node and edge version of react-server-dom-webpack/client. The node version has an implementation for busboy whereas the edge version does not. Next is currently configured to use busboy when handling a fetch action sent as multipart with a node runtime. Ideally we'd only bundle the one platform we are buliding for but some additional refactoring to support better forking is possibly required here
This PR also updates react from 09285d5a7 to d900fadbf.
### React upstream changes
- https://github.com/facebook/react/pull/27439
- https://github.com/facebook/react/pull/26763
- https://github.com/facebook/react/pull/27434
- https://github.com/facebook/react/pull/27433
- https://github.com/facebook/react/pull/27424
- https://github.com/facebook/react/pull/27428
- https://github.com/facebook/react/pull/27427
- https://github.com/facebook/react/pull/27315
- https://github.com/facebook/react/pull/27314
- https://github.com/facebook/react/pull/27400
- https://github.com/facebook/react/pull/27421
- https://github.com/facebook/react/pull/27419
- https://github.com/facebook/react/pull/27418
This PR enables the generation of source maps for the bundled runtimes. This is fast enough that we can just use the optimal source map option directly. This is useful if you're having some errors in Next.js itself.
This swaps the existing minifier for the Next.js runtime bundling from [terser](https://github.com/terser/terser) over to [swc](https://swc.rs/docs/configuration/minification). This small change has slightly decreased development file sizes, and dramatically decreased the time it takes to bundle the runtime. The results below were ran once on my MacBook Pro (M1 Pro).
## Compilation Speed
About 3 times faster
- `next_bundle_pages_dev`: improved from 7.28s to 1.85s.
- `next_bundle_pages_turbo`: improved from 7.28s to 1.85s.
- `next_bundle_pages_prod`: improved from 7.28s to 1.93s.
- `next_bundle_server`: improved from 7.28s to 2.68s.
- `next_bundle_app_turbo`: improved from 8.09s to 2.89s.
- `next_bundle_app_turbo_experimental`: improved from 7.74s to 2.77s.
- `next_bundle_app_prod`: improved from 7.86s to 2.77s.
- `next_bundle_app_dev`: improved from 9.41s to 2.86s.
- `next_bundle_app_prod_experimental`: improved from 8.01s to 3.12s.
- `next_bundle_app_dev_experimental`: improved from 9.11s to 3.58s.
- `next_bundle`: improved from 9.50s to 3.69s.
## File Sizes
About the same, small improvements in development bundle sizes
```shell
# Using terser
692K packages/next/dist/compiled/next-server/app-page-experimental.runtime.dev.js
460K packages/next/dist/compiled/next-server/app-page-experimental.runtime.prod.js
460K packages/next/dist/compiled/next-server/app-page-turbo-experimental.runtime.prod.js
436K packages/next/dist/compiled/next-server/app-page-turbo.runtime.prod.js
672K packages/next/dist/compiled/next-server/app-page.runtime.dev.js
436K packages/next/dist/compiled/next-server/app-page.runtime.prod.js
68K packages/next/dist/compiled/next-server/app-route-experimental.runtime.dev.js
48K packages/next/dist/compiled/next-server/app-route-experimental.runtime.prod.js
48K packages/next/dist/compiled/next-server/app-route-turbo-experimental.runtime.prod.js
48K packages/next/dist/compiled/next-server/app-route-turbo.runtime.prod.js
68K packages/next/dist/compiled/next-server/app-route.runtime.dev.js
48K packages/next/dist/compiled/next-server/app-route.runtime.prod.js
28K packages/next/dist/compiled/next-server/pages-api-turbo.runtime.prod.js
28K packages/next/dist/compiled/next-server/pages-api.runtime.dev.js
28K packages/next/dist/compiled/next-server/pages-api.runtime.prod.js
64K packages/next/dist/compiled/next-server/pages-turbo.runtime.prod.js
68K packages/next/dist/compiled/next-server/pages.runtime.dev.js
64K packages/next/dist/compiled/next-server/pages.runtime.prod.js
204K packages/next/dist/compiled/next-server/server.runtime.prod.js
# Using swc
684K packages/next/dist/compiled/next-server/app-page-experimental.runtime.dev.js
460K packages/next/dist/compiled/next-server/app-page-experimental.runtime.prod.js
460K packages/next/dist/compiled/next-server/app-page-turbo-experimental.runtime.prod.js
436K packages/next/dist/compiled/next-server/app-page-turbo.runtime.prod.js
660K packages/next/dist/compiled/next-server/app-page.runtime.dev.js
436K packages/next/dist/compiled/next-server/app-page.runtime.prod.js
68K packages/next/dist/compiled/next-server/app-route-experimental.runtime.dev.js
48K packages/next/dist/compiled/next-server/app-route-experimental.runtime.prod.js
48K packages/next/dist/compiled/next-server/app-route-turbo-experimental.runtime.prod.js
48K packages/next/dist/compiled/next-server/app-route-turbo.runtime.prod.js
68K packages/next/dist/compiled/next-server/app-route.runtime.dev.js
48K packages/next/dist/compiled/next-server/app-route.runtime.prod.js
28K packages/next/dist/compiled/next-server/pages-api-turbo.runtime.prod.js
28K packages/next/dist/compiled/next-server/pages-api.runtime.dev.js
28K packages/next/dist/compiled/next-server/pages-api.runtime.prod.js
64K packages/next/dist/compiled/next-server/pages-turbo.runtime.prod.js
68K packages/next/dist/compiled/next-server/pages.runtime.dev.js
64K packages/next/dist/compiled/next-server/pages.runtime.prod.js
204K packages/next/dist/compiled/next-server/server.runtime.prod.js
```
## What
This PR changes Next.js to bundle its vendored React libraries so that the App Router pages can use those built-in versions.
## Why
Next.js supports both Pages and App Router and we've gone through a lot of iteration to make sure that Next.js stays flexible wrt to the version of React used: in Pages, we want to use the React provided by the user and in the App Router, to be able to use it, we need to use the canary version of React, which we've built into Next.js for convenience.
The problem stems from the fact that you can't run two different instances of React (by design).
Previously we have a dual worker setup, where we would separate completely each Next.js versions (App and Pages) so that they would not overlap with each other, however this approach was not great performance and memory wise.
We've recently tried using an ESM loader and a single process, but this change would still opt you into the React canary version if you had an app page, which breaks some assumptions.
## How
A list of the changes in this PR:
### New versions of the Next.js runtime
Since we now compile a runtime per type of page (app/route/api/pages), in order to bundle the two versions of React that we vendored, we introduced a new type of bundle suffixed by `-experimental`. This bundle will have the bleeding edge React needed for Server Actions and Next.js will opt you in into that runtime automatically.
For internal contributors, it means that we now run a compiler for 10 subparts of Next.js:
- next_bundle_server
- next_bundle_pages_prod
- next_bundle_pages_turbo
- next_bundle_pages_dev
- next_bundle_app_turbo_experimental
- next_bundle_app_prod
- next_bundle_app_prod_experimental
- next_bundle_app_turbo
- next_bundle_app_dev_experimental
- next_bundle_app_dev
### Simplified require-hook
Since the versions of React are correctly re-routed at build time for app pages, we don't need the require hook anymore
### Turbopack changes
The bundling logic in Turbopack has been addressed to properly follow the new logic
### Changes to the shared contexts system
Some context files need to have a shared instance between the rendering runtime and the user code, like the one that powers the `next/image` component. In general, the aliasing setup takes care of that but we need the require hook for code that is not compiled to reroute to the correct runtime. This only happens for pages node_modules.
A new Turbopack resolving plugin has been added to handle that logic in Turbopack.
### Misc changes
- `runtime-config` (that powers `next/config`) has been converted to an `.external` file, as it should have been
- there are some rules that have been added to the aliases to support the usage of `react-dom/server` in a server-components. We can do that now since the runtime takes care of separating the versions of React.
Co-authored-by: JJ Kasper <22380829+ijjk@users.noreply.github.com>
This PR tweaks the webpack config so that when we compile user code with webpack, we remove all branches of code that are guarded with `process.env.TURBOPACK`. I noticed that the bundle runtime was not tree-shaken correctly, this fixes that.
## What?
In Next, rendering a route involves 3 layers:
- the routing layer, which will direct the request to the correct route to render
- the rendering layer, which will take a route and render it appropriately
- the user layer, which contains the user code
In #51831, in order to optimise the boot time of Next.js, I introduced a change that allowed the routing layer to be bundled. In this PR, I'm doing the same for the rendering layer. This is building up on @wyattjoh's work that initially split the routing and the rendering layer into separate entry-points.
The benefits of having this approach is that this allows us to compartmentalise the different part of Next, optimise them individually and making sure that serving a request is as efficient as possible, e.g. rendering a `pages` route should not need code from the `app router` to be used.
There are now 4 different rendering runtimes, depending on the route type:
- app pages: for App Router pages
- app routes: for App Router route handlers
- pages: for legacy pages
- pages api: for legacy API routes
This change should be transparent to the end user, beside faster cold boots.
## Notable changes
Doing this change required a lot of changes for Next.js under the hood in order to make the different layers play well together.
### New conventions for externals/shared modules
The big issue of bundling the rendering runtimes is that the user code needs to be able to reference an instance of a module/value created in Next during the render. This is the case when the user wants to access the router context during SSR via `next/link` for example; when you call `useContext(value)` the value needs to be the exact same reference to one as the one created by `createContext` earlier.
Previously, we were handling this case by making all files from Next that were affected by this `externals`, meaning that we were marking them not to be bundled.
**Why not keep it this way?**
The goal of this PR as stated previously was to make the rendering process as efficient as possible, so I really wanted to avoid extraneous fs reads to unoptimised code.
In order to "fix" it, I introduced two new conventions to the codebase:
- all files that explicitly need to be shared between a rendering runtime and the user code must be suffixed by `.shared-runtime` and exposed via adding a reference in the relevant `externals` file. At compilation time, a reference to a file ending with this will get re-written to the appropriate runtime.
- all files that need to be truly externals need to be suffixed by `.external`. At compilation time, a reference to it will stay as-is. This special case is needed mostly only for the async local storages that need to be shared with all three layers of Next.
As a side effect, we should be bundling more of the Next code in the user bundles, so it should be slightly more efficient.
### App route handlers are compiled on their own layer
App route handlers should be compiled in their own layer, this allows us to separate more cleanly the compilation logic here (we don't need to run the RSC logic for example).
### New rendering bundles
We now generate a prod and a dev bundle for:
- the routing server
- the app/pages SSR rendering process
- the API routes process
The development bundle is needed because:
- there is code in Next that relies on NODE_ENV
- because we opt out of the logic referencing the correct rendering runtime in dev for a `shared-runtime` file. This is because we don't need to and that Turbopack does not support rewriting an external to something that looks like this `require('foo').bar.baz` yet. We will need to fix that when Turbopack build ships.
### New development pipeline
Bundling Next is now required when developing on the repo so I extended the taskfile setup to account for that. The webpack config for Next itself lives in `webpack.config.js` and contains the logic for all the new bundles generated.
### Misc changes
There are some misc reshuffling in the code to better use the tree shaking abilities that we can now use.
fixes NEXT-1573
Co-authored-by: Alex Kirszenberg <1621758+alexkirsz@users.noreply.github.com>
