Continue the work to improve #32314.
This PR splits `webpack-build` into an `index` module and an `impl`
module. The `index` module invokes `impl` based on that if it should use
workers or in the same process.
Another notable change is that I modified the number of static workers,
to make it also based on remaining memory. I did some benchmarks and
found that using the number of CPUs might not be ideal. After the build
process (if `webpackBuildWorker` **isn't** enabled), the memory usage
was already very high. While the memory usage of the main process didn't
increase in the static optimization step, the overall number went high
again because we created 10 static workers:
As of today, a MacBook Pro has 10 CPU cores and 16 GB memory. The idle
memory of the system might be 8 GB, and the remaining memory when
starting the static workers might be 4~6 GB only. Here's a benchmark of
`next build` on `vercel-site`:
The memory dropped there quickly because it has `webpackBuildWorker`
enabled, but then it increased again with static workers (note that we
already have `cpus: Math.round(require('node:os').cpus().length / 2) ||
1` configured in that repo). It makes more sense to limit that inside
Next.js natively.
- Add links to the error warning
- Log with warning prefix for every url that missing `metadataBase`, so users can easily locate where it's from
Closes NEXT-931
### What?
Refactors the resolving logic for the `app` loader tree. This PR ensures
it's used to create entrypoints in turbopack. Next up is integrating it
into the webpack build too.
These changes also ensure that parallel routes resolving is applied,
which previously wasn't supported in turbopack.
### Why?
Part of the effort to deduplicate critical logic between
turbopack/webpack in Next.js, this will help land features in
turbopack/webpack at the same time.
### How?
Quite a few changes. @sokra helped a ton on this PR. `app_structure.rs`
was changed to the new resolving logic so most of the logic is there.
Additionally we added support for calling the same function in two ways
from Node.js: `turbo.entrypoints.get` and `turbo.entrypoints.getStream`.
`get` can be used by `next build` to get the full list of
entrypoints/loadertrees once. `getStream` has watching built-in and
calls a callback function with the new list anytime a file is added that
would change the loadertree.
<!-- Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
## For Contributors
### Improving Documentation or adding/fixing Examples
- The "examples guidelines" are followed from our contributing doc
https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md
- Make sure the linting passes by running `pnpm build && pnpm lint`. See
https://github.com/vercel/next.js/blob/canary/contributing/repository/linting.md
### Fixing a bug
- Related issues linked using `fixes #number`
- Tests added. See:
https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md
### Adding a feature
- Implements an existing feature request or RFC. Make sure the feature
request has been accepted for implementation before opening a PR. (A
discussion must be opened, see
https://github.com/vercel/next.js/discussions/new?category=ideas)
- Related issues/discussions are linked using `fixes #number`
- e2e tests added
(https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Documentation added
- Telemetry added. In case of a feature if it's used or not.
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md
## For Maintainers
- Minimal description (aim for explaining to someone not on the team to
understand the PR)
- When linking to a Slack thread, you might want to share details of the
conclusion
- Link both the Linear (Fixes NEXT-xxx) and the GitHub issues
- Add review comments if necessary to explain to the reviewer the logic
behind a change
### What?
### Why?
### How?
Closes NEXT-
Fixes #
-->
Should fill dynamic routes url with params when generate the metadata
image urls.
For `/(group)`, we need to normalize the path first;
For dynamic routes `/[slug]`, we need to fill the params.
Change the image module from `() => image` to `(prop) => image(proop)`
so we can fill the runtime `params` into url
Closes NEXT-932
---------
This updates to have a separate routing process and separate rendering
processes for `pages` and `app` so that we can properly isolate the two
since they rely on different react versions.
Besides allowing the above mentioned isolation this also helps us
control recovering from process crashes easier as pieces are more
isolated from one another e.g. an infinite loop during rendering will no
longer block the compiler and can be stopped/restarted as needed.
In follow-up PRs we will continue to separate out the routing logic from
the rendering logic so that each process only loads what is relevant to
it helping simplify the flow for requests regardless of type.
---------
Co-authored-by: Shu Ding <g@shud.in>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
### What / why?
Currently there is a bug because the `subTreeData` can only be applied
once, however, we reuse the same response when a url was prefetched.
During the first render of the response we apply the cacheNodes and then
these should be reused for subsequent navigations. There's currently a
bug because `fillLazyItemsTillLeafWithHead` creates `LAZY_INITIALIZED`
cache nodes that don't copy the previous cache node `parallelRoutes` and
`subTreeData` value, causing the cache nodes below where the
`subTreeData` is applied to be deleted from the cache, even when they're
not affected by the navigation.
### How?
This PR adds copying of the cache node when it exists already, I've
added a marker for `wasPrefetched` to trigger that behavior but in
talking to @feedthejim about this it seems we can swap to this behavior
always when applying the server response, that needs to be discussed
further though, this is temporary implementation with the fix for routes
that were prefetched.
Fixes NEXT-402
<!-- Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
## For Contributors
### Improving Documentation or adding/fixing Examples
- The "examples guidelines" are followed from our contributing doc
https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md
- Make sure the linting passes by running `pnpm build && pnpm lint`. See
https://github.com/vercel/next.js/blob/canary/contributing/repository/linting.md
### Fixing a bug
- Related issues linked using `fixes #number`
- Tests added. See:
https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md
### Adding a feature
- Implements an existing feature request or RFC. Make sure the feature
request has been accepted for implementation before opening a PR. (A
discussion must be opened, see
https://github.com/vercel/next.js/discussions/new?category=ideas)
- Related issues/discussions are linked using `fixes #number`
- e2e tests added
(https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Documentation added
- Telemetry added. In case of a feature if it's used or not.
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md
## For Maintainers
- Minimal description (aim for explaining to someone not on the team to
understand the PR)
- When linking to a Slack thread, you might want to share details of the
conclusion
- Link both the Linear (Fixes NEXT-xxx) and the GitHub issues
- Add review comments if necessary to explain to the reviewer the logic
behind a change
### What?
### Why?
### How?
Closes NEXT-
Fixes #
-->
---------
Co-authored-by: JJ Kasper <jj@jjsweb.site>
Since both `exportPage` and `isPageStatic` methods in the static
optimization worker involve loading the actual page component, it's
better to assign the same page to the same worker for these 2 tasks to
avoid unnecessary memory allocation.
Co-authored-by: JJ Kasper <jj@jjsweb.site>
This ensures we properly honor the `export const fetchCache` config and
also ensures we properly bypass fetch-cache when an On-Demand
Revalidation is occurring.
The `export const dynamic` handling is not changed here as that was
behaving correctly and should not influence fetch cache handling only
whether a page is prerendered fully or treated as SSR.
Fixes: https://github.com/vercel/next.js/issues/47273
x-ref: [slack
thread](https://vercel.slack.com/archives/C042LHPJ1NX/p1679078572123979)
Fix the module resolving error found during development, because now few
app route code is bundled into RSC server layer, it can't resolve
opentelementry properly and will throw the error blow due to the check
of `require('@opentelemetry/api')` to see if it's installed.
```
Module not found: Can't resolve '@opentelemetry/api'
```
Thus I alias it to default `next/dist/compiled/@opentelemetry/api` when
it doesn't have external module
---------
For backward compatibility, we only handle `favicon.ico` file to
generate `/favicon.ico` route and link tag. If you want to use other
extension such as `png`, use `icon(\d)?.[ext]`
Fixes#46993.
The issue is introduced in #46705 by @ijjk
Next.js uses `jest-worker` for multiprocessing. `jest-worker` supports both `child_process` and `worker_threads` modes. Next.js use `child_process` mode by default, but can also switch to using `worker_threads` mode with an experimental flag `config.experimental.workerThreads`.
In #46705, @ijjk applies a fix that works for the `child_process` mode, which breaks the `worker_threads` mode (`jest-worker`'s `NodeThreadWorker` interface doesn't have the private property `_child`), causing static HTML export to fail with the following error (#46993):
```
> Build error occurred
TypeError: Cannot read properties of undefined (reading 'on')
at createWorker (/Users/[redacted]/node_modules/next/dist/lib/worker.js:32:31)
at new Worker (/Users/[redacted]/node_modules/next/dist/lib/worker.js:42:9)
at /Users/[redacted]/node_modules/next/dist/build/index.js:629:35
at async Span.traceAsyncFn (/Users/[redacted]/node_modules/next/dist/trace/trace.js:79:20)
at async Object.build [as default] (/Users/[redacted]/node_modules/next/dist/build/index.js:74:29)
```
The PR fixes that.
This ensures we don't normalize locales in the URL with the
skipMiddlewareUrlNormalize flag enabled so that casing redirects can be
applied correctly for locales.
Turbopack starts up the router process with all ENV values, but the edge
function definition didn't list any `env` keys for the function
invocation. So, middleware couldn't access any ENV values.
Turbopack doesn't currently have a way to determine what ENV keys are
actually used by the source program, so I'm just passing everything
defined. I'm not sure if that's an issue during dev (I could see it
being one for the build process, but that doesn't matter for this case).
Fixes#47766
Fixes WEB-831
Fixes WEB-834
This ensures we check if a path is `appRouter` during prefetching so
that we can trigger hard navigations quicker when routing from pages ->
app. An additional config is also exposed to allow configuring the
potential false positive rate for the client filter.
x-ref: [slack
thread](https://vercel.slack.com/archives/C017QMYC5FB/p1680225393243459)
Fixes an issue with newer app directory usage:
```
Failed to compile.
../../node_modules/next/dist/server/future/route-handlers/app-route-route-handler.d.ts:11:15
Type error: ',' expected.
```
---------
Allow to use relative paths which starting with dot (e.g. `./[paths]`) for urls under `metadata.alternates`.
This allow user to simplify setting `canonical` or other such like `languages` once in root layout with `metadataBase` and a simple relative path, then next.js will resolve it with current pathname of the page.
For example
```js
export const metadata = {
metadataBase: new URL('https://mydomain.com'),
alternates: {
canonical: './'
}
}
```
Then:
for page `/` it will generate `https://mydomain.com`;
for page `/about` it will generate `https://mydomain.com/about`
as your cononical url
Closes NEXT-897
### Minor changes
- always remove trailing slash for `URL.href`
## What
This fix serves to address issues where multiple `Set-Cookie` headers
were combined in some runtimes.
## Why
This is because `set-cookie` behaves differently than other headers in
some cases.
Eg. when iterating on a `Headers` instance, multiple set-cookie headers
are folded. To set them correctly, we need to split them. But it'd not
be enough to naively split on the first occurrence, because `,` is a
valid cookie value when for example it's used in `Expires` in a date
string.
So we use a method to correctly detect where to split the cookie.
This should fix all runtimes.
Note, the spec now has `Headers#getSetCookie` which should be preferred
if it's present. https://github.com/whatwg/fetch/pull/1346. We are using
the [`edge-runtime`](https://github.com/vercel/edge-runtime), so this
should be fixed upstream and then reused in Next.js in the future.
## How
Wherever we can, we reuse the `fromNodeHeaders` and `toNodeHeaders`
methods that have the correct implementation. This should be preferred
in the future in other parts of the codebase. We fixed some related TS
issues as well.
Fixes#46579, supersedes #40579
fix NEXT-735 ([link](https://linear.app/vercel/issue/NEXT-735))
---------
Co-authored-by: Balázs Orbán <info@balazsorban.com>
### What?
This PR adds an internal compile time flags for the `next-swc`, which exposes two runtime apis into next.js to initialize heap-profiling enabled memory allocator and teardown those once next.js exits. While there are newly added 2 js interfaces (`initHeapProfiler`, `teardownHeapProfiler`) underlying napi binary itself have compile time flags to actually enable those feature: any user who runs npm-published next.js cannot enable this features with any kind of runtime configuration. Only manually built next-swc binary with specific flag can enable this. Since this is primarily for the CI testing workflow only, those flag / configs are not visibily exposed as well.
### Why?
It is for some experiments on the CI to see if it can observe some of memory pressure issues (WEB-593, WEB-804) while it is not easily reproducible on the local machines.
Based on some suggestions from @sokra, this serves to unify the Route Handlers and Route Modules to provide a unified interface for defining route modules and handlers and defining their dependancies.
Adding default metadata before causes duplicated metadata tags when
adding server inserted html. Remove the ones in the not found boundary
Now we use same way to render metadata to render the default metadata in
error html
Fixes the issue introduced in #47404
---------
Co-authored-by: JJ Kasper <jj@jjsweb.site>
### What?
- Provide a default `metadataBase`
- Always resolve urls that could be resolved as absolute with
`metadataBase`, e.g. tw/og urls, canonical urls
- Give a warning in dev mode if user doesn't provide one in dev
- Error if you don't have it but it's required in production
On production it will leverage `VERCEL_URL` if users expose it to the
deployment
### Why?
OG image urls are required to be absolute urls instead of relative urls.
For metadata image conventions we let users don't have to provide
`metadataBase` explicitly when they expect it should be the origin of
their next app.
### How?
Closes NEXT-887
---------
### What?
Followup for #47630 , use ts namespace to group MetadataRoute.
`MetadataRoute['xxx']` -> `MetadataRoute.xxx`
### Why?
namespace is convenient to write for these grouped types such as
`MetadataRoute.Robots`. And `MetadataRoute` itself is not actually a
type. So it can be conveniently typed by ts users.
### How?
Closes NEXT-912
---------
just took a bit of banging my head against the wall 👀
the `'dynamic route/page check'` route resolves `fsRoutes` (which leads to an object with `{pages: undefined}` because the route doesn't have the list of pages(?))
that route only gets enabled if you have fallback rewrites though
### What?
- Add not found boundary to app router
- Move `head` cache back to app router
### Why?
We want the head to be rendered separately from body, previously to be
able to use `redirect()` and `notFound()` in `generateMetadata` we move
the head cache into layout-router to be wrapped by not found and
redirect boundaries. Since redirect boudary is already moved to
app-router, so we only need to add not found boundary and move head
cache to app router.
Notice: there's a limitation that we can't find the corresponding not
found of page if you throw notFound in generateMetadata, the root layout
+ root/default not found will be used to generate the 404 page
### How?
Closes NEXT-864
Fixes#46738
fix NEXT-888 ([link](https://linear.app/vercel/issue/NEXT-888))
---------
Previously, `next start` was incorrectly trying to optimize an empty `src` prop.
This PR fixes the behavior so that empty `src` will automatically set `unoptimized` (same behavior as data urls).
fix NEXT-915 ([link](https://linear.app/vercel/issue/NEXT-915))
Fixed issue where `require` will return a Promise if (and only if) there
is an ESM module imported in `instrumentation.ts`:
61cd219f15/packages/next/src/server/next-server.ts (L212)
Had to move stuff to async function from constructor, but luckily we
already have `prepare`.
### What / Why / How?
In a project with `"type": "module"` in `package.json` and a `compilerOptions.moduleResolution` of `Node16` or `NodeNext` in `tsconfig.json`, running `tsc` will yield errors like the following:
```
.next/types/app/[...not_found]/page.ts:1:40 - error TS2307: Cannot find module 'next/dist/lib/metadata/types/metadata-interface' or its corresponding type declarations.
1 import type { ResolvingMetadata } from 'next/dist/lib/metadata/types/metadata-interface';
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.next/types/app/layout.ts:2:24 - error TS2835: Relative import paths need explicit file extensions in EcmaScript imports when '--moduleResolution' is 'node16' or 'nodenext'. Did you mean '../../../app/layout.jsx'?
2 import * as entry from '../../../app/layout'
~~~~~~~~~~~~~~~~~~~~~
.next/types/app/layout.ts:3:40 - error TS2307: Cannot find module 'next/dist/lib/metadata/types/metadata-interface' or its corresponding type declarations.
3 import type { ResolvingMetadata } from 'next/dist/lib/metadata/types/metadata-interface'
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
These files are checked because of the following `tsconfig.json` line which Next.js adds:
```diff
{
"include": [
+ ".next/types/**/*.ts",
]
}
```
Adding the extensions will make it work with these projects and still maintain backwards compatibility with other TypeScript / module configurations.
### What?
This PR vendors @vercel/og and export `ImageResponse` from
`next/server`. When you render a opengraph image the below code snippets
will be legit:
```tsx
import { ImageResponse } from 'next/server'
export default function og() {
return new ImageResponse(<div style={{ width: '100%', height: '100%' }}>hello</div>)
}
```
### Why?
To make development more easier, user can directly use `@vercel/og`
Image Response with nextjs instead of install it and use it. This makes
building metadata icons, og or twitter images more convenient.
### How?
Closes NEXT-899
