This strips the internal routing headers added by the routing server. Based on some of the changes being introduced in https://github.com/vercel/next.js/pull/54813 this strips the headers, but this also adds some internal flags to turn this off during testing to validate correct routing beheviour.
This is to fix an issue where these redirect side effects can be fired multiple times when the router reducer state changes. This block is still run when the router state updates, which can lead to superfluous attempts to redirect to a page.
With these changes, we keep track of the page that is being redirected to. If a re-render occurs while that request is in flight, we don't trigger the side effects.
[Slack x-ref](https://vercel.slack.com/archives/C04DUD7EB1B/p1694049914264839)
### What?
Upgrade TypeScript to the latest version as of this PR. **This does not affect users, as the change is only for our repository.**
### Why?
Part of some upcoming PRs to try to clean up cookie handling, now that `getSetCookie` is available. Since we use `undici`, which [implements it](https://github.com/nodejs/undici/pull/1915), we can get rid of some code to rely more on the platform.
This PR is needed to get the types for `Headers#getSetCookie` which was added in 5.2
### How?
I needed to update some dependency types to get build to pass, but other than that, only needed to bump from `5.1.6` to `5.2.2`, so hopefully all is fine.
The previous RegExp for data routes when i18n was enabled yielded a pattern like:
```
^\/_next\/data\/development\/(?<nextLocale>.+?)\/about.json$
^\/_next\/data\/development\/(?<nextLocale>.+?)\/blog/about.json$
```
But the capture group for the `nextLocale` did so greedily, where the following:
```
/_next/data/development/en-US/blog/about.json
```
Would actually match both routes.
This changes it to prevent the locale from including a `/` via `[^/]`, resulting in the new expressions:
```
^\/_next\/data\/development\/(?<nextLocale>[^/]+?)\/about.json$
^\/_next\/data\/development\/(?<nextLocale>[^/]+?)\/blog/about.json$
```
## What?
In Next, rendering a route involves 3 layers:
- the routing layer, which will direct the request to the correct route to render
- the rendering layer, which will take a route and render it appropriately
- the user layer, which contains the user code
In #51831, in order to optimise the boot time of Next.js, I introduced a change that allowed the routing layer to be bundled. In this PR, I'm doing the same for the rendering layer. This is building up on @wyattjoh's work that initially split the routing and the rendering layer into separate entry-points.
The benefits of having this approach is that this allows us to compartmentalise the different part of Next, optimise them individually and making sure that serving a request is as efficient as possible, e.g. rendering a `pages` route should not need code from the `app router` to be used.
There are now 4 different rendering runtimes, depending on the route type:
- app pages: for App Router pages
- app routes: for App Router route handlers
- pages: for legacy pages
- pages api: for legacy API routes
This change should be transparent to the end user, beside faster cold boots.
## Notable changes
Doing this change required a lot of changes for Next.js under the hood in order to make the different layers play well together.
### New conventions for externals/shared modules
The big issue of bundling the rendering runtimes is that the user code needs to be able to reference an instance of a module/value created in Next during the render. This is the case when the user wants to access the router context during SSR via `next/link` for example; when you call `useContext(value)` the value needs to be the exact same reference to one as the one created by `createContext` earlier.
Previously, we were handling this case by making all files from Next that were affected by this `externals`, meaning that we were marking them not to be bundled.
**Why not keep it this way?**
The goal of this PR as stated previously was to make the rendering process as efficient as possible, so I really wanted to avoid extraneous fs reads to unoptimised code.
In order to "fix" it, I introduced two new conventions to the codebase:
- all files that explicitly need to be shared between a rendering runtime and the user code must be suffixed by `.shared-runtime` and exposed via adding a reference in the relevant `externals` file. At compilation time, a reference to a file ending with this will get re-written to the appropriate runtime.
- all files that need to be truly externals need to be suffixed by `.external`. At compilation time, a reference to it will stay as-is. This special case is needed mostly only for the async local storages that need to be shared with all three layers of Next.
As a side effect, we should be bundling more of the Next code in the user bundles, so it should be slightly more efficient.
### App route handlers are compiled on their own layer
App route handlers should be compiled in their own layer, this allows us to separate more cleanly the compilation logic here (we don't need to run the RSC logic for example).
### New rendering bundles
We now generate a prod and a dev bundle for:
- the routing server
- the app/pages SSR rendering process
- the API routes process
The development bundle is needed because:
- there is code in Next that relies on NODE_ENV
- because we opt out of the logic referencing the correct rendering runtime in dev for a `shared-runtime` file. This is because we don't need to and that Turbopack does not support rewriting an external to something that looks like this `require('foo').bar.baz` yet. We will need to fix that when Turbopack build ships.
### New development pipeline
Bundling Next is now required when developing on the repo so I extended the taskfile setup to account for that. The webpack config for Next itself lives in `webpack.config.js` and contains the logic for all the new bundles generated.
### Misc changes
There are some misc reshuffling in the code to better use the tree shaking abilities that we can now use.
fixes NEXT-1573
Co-authored-by: Alex Kirszenberg <1621758+alexkirsz@users.noreply.github.com>
### What?
There are tests under `next-dev-tests` which used native binary to run tests for Turbopack. This should belong to next.js integration tests, and also indeed there are overlaps.
As a first step, PR removes duplicated styled-jsx test and mark existing test under turbopack test filter as enabled.
Closes WEB-1510
nonce's are limited to characters found in base64 encoding, uuids contain '-' which breaks the spec,
converting to a base64 string after generating simplifies this
---
This was a bit gotcha in our project, there are a few tools that only expect there to be a single `-` and do a split based off it (so when there are >1 they fail)
## Rules for nonce's
- The nonce must be unique for each HTTP response
- The nonce should be generated using a cryptographically secure random generator
- The nonce should have sufficient length, aim for at least 128 bits of entropy (32 hex characters, or about 24 base64 characters).
- Script tags that have a nonce attribute must not have any untrusted / unescaped variables within them.
- The characters that can be used in the nonce string are limited to the characters found in base64 encoding.
This adds session ids and anonymous ids to trace metadata.
Note: This required setting `session`id on the `Telemetry` object as a public property. Set as readonly.
Closes WEB-1500
This moves `resolve-href` into `next/src/client` to make sure that when it calls `normalizeTrailingSlash`, that function has access to `process.env.__NEXT_MANUAL_TRAILING_SLASH` (inlined via `DefinePlugin`).
Closes NEXT-1599
Fixes#54984
### What?
Switch the default for `--turbo` to the new `--experimental-turbo`, remove the old code in next.js
### Why?
The new approach will be used in future
Closes WEB-1506
This PR will enable minifying the *server* part of the user code by default when running `next build`.
## Explanation
Next.js compiles two versions of your code: the client version for the app that runs in the browser, and the server for the code that will run on the server. Whilst the client code has always been minified and optimised, we haven't done so historically for the server side.
## How does this impact me?
There are several consequences to this change:
- minified code makes error stacks less readable. To fix that, you can use source maps, which you can enable with `experimental.serverSourceMaps`. This is not enabled by default as this is an expensive operation.
- however the server code will be optimised for size so as a result, cold boots should improve
## opting out
you can opt out via specifying `experimental.serverMinification: false` in `next.config.js`
In dev mode, we're using a catch-all route for dynamic metadata routes, e.g. page path `/twitter-image` would become `/twitter-image/[[...metadata_id...]]/route` as a dynamic custom app route.
But we're missing to convert it in filesystem scanning for routing purpose, adding the metadata related normalizing logic for app page to align with other places.
This PR introduces a change in the Next.js server that should improve memory usage nicely during dev.
## How
While investigating the repro cases in #54708, @timneutkens and I noticed that the high memory usage often involved `googleapis`. Digging a bit more, I also saw that in dev, the bundle generated for a page using `googleapis` was around 80MB and that requiring it in Node.js increased memory by 160MB 🥲 and that requiring another page that also used this also increased the memory by 160MB.
The problem is that Next.js, on new navigations and hot reloads, might need to load/reload all the code required for the page *all the time*. This issue is also exacerbated by the fact that there's a nasty Node.js bug that makes it impossible to clean that memory up, leading to memory bloat and overall a pretty bad DX.
So if we can't clean this up, what can we do about it?
The change I'm introducing in this PR is that I'm changing Next.js in order to split the code you're using from `node_modules` from the code you've written in different chunks. The idea is that the heavy code you're loading from `node_modules` is only gonna be loaded once per session this time.
This should make a navigation/page reload only load the user bundle now. On my simple test case, the cost of navigation went from ~200MB to ~40MB.
A few notes on the implementation:
- The chunks for the `node_modules` are split at the module level, this is to ensure that each of the node_modules dependencies is split in the most memory efficient manner. If it was a big monolithic chunk, it would potentially be reloaded again and again whenever reloaded, leading to leakage.
- I'm guessing we could do the same for the Edge server
- the first load for a page will still be fairly heavy memory wise, there's probably something else we can do there
- there's also an issue with the webpack require cache being flushed, whereas it could be reused
## comparisons
### navigating a page
before
<img width="284" alt="CleanShot 2023-09-04 at 21 00 46@2x" src="https://github.com/vercel/next.js/assets/11064311/44e37df8-4414-4ca1-b6bf-fb0fb11751ea">
after
<img width="392" alt="CleanShot 2023-09-04 at 20 58 53@2x" src="https://github.com/vercel/next.js/assets/11064311/46226123-a73a-4132-a99d-fb812e59df46">
When using page routes we need to pass the contents of the `export const config` to the `extraConfig` populating the functions config manifest rather then relying on named exports like in app router.
### 🧐 What's in there?
As discussed with @huozhi, it's restoring code from #52393 which I forgot to restore in #54891.
It is unlikely to make a difference, but we'll set globalWindow based on the desired test environment. This will not make any difference for most people.
### 🧪 How to test?
Once you've rebuilt your local version of Next, `pnpm testheadless jest/rsc`
Co-authored-by: Jiachi Liu <4800338+huozhi@users.noreply.github.com>
This bumps `undici` to the latest version to match the one used in newer Node.js versions', potentially fixing compatibility issues.
I also cleaned up some related `global as any` usage.
Ports content from the Pages and Layouts section about templates and formats to match other API reference pages.
Co-authored-by: Michael Novotny <446260+manovotny@users.noreply.github.com>
While investigating the HMR event types I noticed `pong` is not being used in Pages Router nor in App Router.
This PR removes the code that sends it as it's essentially dead code.
Adds `serverError` to the list of HMR actions, missed this in the earlier PRs as it wasn't using `hotReloader.send`.
Also ensures on-demand-entry-handler reuses the hotReloader.send method.
Ensures only one name is used for the event, makes it easier to search for, going to replace this with the constant like the other events in a follow-up PR but this can be landed as-is.
