## What
This PR changes Next.js to bundle its vendored React libraries so that the App Router pages can use those built-in versions.
## Why
Next.js supports both Pages and App Router and we've gone through a lot of iteration to make sure that Next.js stays flexible wrt to the version of React used: in Pages, we want to use the React provided by the user and in the App Router, to be able to use it, we need to use the canary version of React, which we've built into Next.js for convenience.
The problem stems from the fact that you can't run two different instances of React (by design).
Previously we have a dual worker setup, where we would separate completely each Next.js versions (App and Pages) so that they would not overlap with each other, however this approach was not great performance and memory wise.
We've recently tried using an ESM loader and a single process, but this change would still opt you into the React canary version if you had an app page, which breaks some assumptions.
## How
A list of the changes in this PR:
### New versions of the Next.js runtime
Since we now compile a runtime per type of page (app/route/api/pages), in order to bundle the two versions of React that we vendored, we introduced a new type of bundle suffixed by `-experimental`. This bundle will have the bleeding edge React needed for Server Actions and Next.js will opt you in into that runtime automatically.
For internal contributors, it means that we now run a compiler for 10 subparts of Next.js:
- next_bundle_server
- next_bundle_pages_prod
- next_bundle_pages_turbo
- next_bundle_pages_dev
- next_bundle_app_turbo_experimental
- next_bundle_app_prod
- next_bundle_app_prod_experimental
- next_bundle_app_turbo
- next_bundle_app_dev_experimental
- next_bundle_app_dev
### Simplified require-hook
Since the versions of React are correctly re-routed at build time for app pages, we don't need the require hook anymore
### Turbopack changes
The bundling logic in Turbopack has been addressed to properly follow the new logic
### Changes to the shared contexts system
Some context files need to have a shared instance between the rendering runtime and the user code, like the one that powers the `next/image` component. In general, the aliasing setup takes care of that but we need the require hook for code that is not compiled to reroute to the correct runtime. This only happens for pages node_modules.
A new Turbopack resolving plugin has been added to handle that logic in Turbopack.
### Misc changes
- `runtime-config` (that powers `next/config`) has been converted to an `.external` file, as it should have been
- there are some rules that have been added to the aliases to support the usage of `react-dom/server` in a server-components. We can do that now since the runtime takes care of separating the versions of React.
Co-authored-by: JJ Kasper <22380829+ijjk@users.noreply.github.com>
Only output app routes if there's no pages routes in next build output
For hello world app you'll only see app dir routes
```
Route (app) Size First Load JS
+ First Load JS shared by all 78.1 kB
├ chunks/main-app-e4c0616da69beffe.js 76.5 kB
└ chunks/webpack-bf1a64d1eafd2816.js 1.61 kB
○ (Static) automatically rendered as static HTML (uses no initial props)
```
Also filter out `/favicon.ico` static route as it's confusing
Optimizes how we handle cache tags for soft tags (auto-added by Next.js)
and normal tags (added manually) and adds differentiating between
`revalidatePath('/blog/first')` and page/layout.
Soft tags are not stored across cache entry and instead auto sent along
when checking cache entries. This allows us to prevent storing
exponential amounts of tags across cache entries while still having the
relationship between them so that single path revalidation can work
properly.
x-ref: [slack
thread](https://vercel.slack.com/archives/C042LHPJ1NX/p1690586837903309)
### What?
When running Next in standalone mode, `process.env` is not made
available to the render workers, making it impossible to access
environment variables that aren't provided in `.env` files.
### Why?
`initialEnv` is undefined in `createWorkers` when the server is started
in standalone mode.
### How?
This initializes the workers with `process.env` in case `initialEnv` is
unavailable, similar to the behavior of `loadEnvConfig()`
Closes NEXT-1508
Fixes#53367
Follow up to https://github.com/vercel/next.js/pull/54081 -- this was
restoring the router tree improperly causing an error on bfcache hits
Had to override some default behaviors to prevent `forward` / `back` in
playwright from hanging indefinitely since no load event is firing in
these cases
Fixes#54184
Closes NEXT-1528
Reverts vercel/next.js#53578
This PR (#53578) will break client components test, revert it for now.
Can repro by adding `"use client"` to `app/page.tsx` in `test/production/jest/server-only.test.ts`
```
FAIL app/page.test.jsx
● Test suite failed to run·
Cannot find module 'private-next-rsc-mod-ref-proxy' from 'app/page.jsx'·
Require stack:
```
When an mpa navigation takes place, we currently push the user to the new route and suspend the page indefinitely (x-ref: #49058). When navigating back, if the browser opts into using the [bfcache](https://web.dev/bfcache/), it will remain suspended and `pushRef.mpaNavigation` will be true. This means that anything that would cause the component to re-render will trigger the mpa navigation again (such as hovering over another `Link`, as reported in #53347)
This PR checks to see if bfcache is being used by observing `PageTransitionEvent.persisted` and if so, resets the router state to clear out `pushRef`.
Closes NEXT-1511
Fixes#53347
### 🧐 What's in there?
This is another attempt to allow testing server-only code with Jest.
### 🧪 How to test?
There's an integration tests which can be triggered with `pnpm testheadless server-only`
Here is a more comprehensive setup:
<details>
<summary><code>app/lib/index.ts</code></summary>
```ts
import 'server-only'
export function add(num1: number, num2: number) {
return num1 + num2
}
```
</details>
<details>
<summary><code>app/lib/index.test.ts</code></summary>
```ts
import { add } from '.'
it('adds two numbers', () => {
expect(add(1, 3)).toEqual(4)
})
```
</details>
<details>
<summary><code>app/client-component.tsx</code></summary>
```ts
'use client'
import { useState } from 'react'
export default function ClientComponent() {
const [text, setText] = useState('not clicked yet')
return <button onClick={() => setText('clicked!')}>{text}</button>
}
```
</details>
<details>
<summary><code>app/client-component.test.tsx</code></summary>
```ts
import { fireEvent, render, screen } from '@testing-library/react'
import ClientComponent from './client-component'
it('can be clicked', async () => {
render(<ClientComponent />)
const button = screen.getByRole('button')
expect(button).toHaveTextContent('not clicked yet')
await fireEvent.click(button)
expect(button).toHaveTextContent('clicked!')
})
```
</details>
<details>
<summary><code>app/server-component.tsx</code></summary>
```ts
import { add } from '@/lib'
export default function ServerComponent({ a, b }: { a: number; b: number }) {
return (
<code role="comment">
{a} + {b} = {add(a, b)}
</code>
)
}
```
</details>
<details>
<summary><code>app/server-component.test.tsx</code></summary>
```ts
import { render, screen } from '@testing-library/react'
import ServerComponent from './server-component'
it('renders', () => {
render(<ServerComponent a={2} b={3} />)
expect(screen.getByRole('comment')).toHaveTextContent('2 + 3 = 5')
})
```
</details>
<details>
<summary><code>app/page.tsx</code></summary>
```ts
import Link from 'next/link'
import ClientComponent from './client-component'
import ServerComponent from './server-component'
export default function Page() {
return (
<>
<h1>Hello World</h1>
<Link href="/dave">Dave?</Link>
<p>
<ClientComponent />
</p>
<p>
<ServerComponent a={5} b={2} />
</p>
</>
)
}
```
</details>
<details>
<summary><code>app/page.test.tsx</code></summary>
```ts
import { render, screen } from '@testing-library/react'
import Page from './page'
it('greets', () => {
render(<Page />)
expect(screen.getByRole('link')).toHaveTextContent('Dave?')
expect(screen.getByRole('heading')).toHaveTextContent('Hello World')
expect(screen.getByRole('button')).toHaveTextContent('not clicked yet')
expect(screen.getByRole('comment')).toHaveTextContent('5 + 2 = 7')
})
```
</details>
<details>
<summary><code>app/[blog]/page.tsx</code></summary>
```ts
import { Metadata } from 'next'
import Link from 'next/link'
type Props = {
params: { blog: string }
}
export async function generateMetadata({
params: { blog: title },
}: Props): Promise<Metadata> {
return { title, description: `A blog post about ${title}` }
}
export default function Page({ params }: Props) {
return (
<>
<div>
<Link href="/">Back</Link>
</div>
<h1>All about {params.blog}</h1>
</>
)
}
```
</details>
<details>
<summary><code>app/[blog]/page.test.tsx</code></summary>
```ts
import { render, screen } from '@testing-library/react'
import Page from './page'
it('has the appropriate title', () => {
const title = 'Jane'
render(<Page params={{ blog: title }} />)
expect(screen.getByRole('heading')).toHaveTextContent(`All about ${title}`)
expect(screen.getByRole('link')).toHaveTextContent('Back')
})
```
</details>
<details>
<summary><code>app/layout.tsx</code></summary>
```ts
export default function RootLayout({ children }) {
return (
<html lang="en">
<body>{children}</body>
</html>
)
}
```
</details>
<details>
<summary><code>jest.config.js</code></summary>
```ts
const nextJest = require('next/jest')
const createJestConfig = nextJest({ dir: './' })
module.exports = createJestConfig({
testEnvironment: 'jsdom',
setupFilesAfterEnv: ['<rootDir>/test-setup.ts'],
})
```
</details>
<details>
<summary><code>package.json</code></summary>
```ts
{
"name": "rsc-test",
"version": "0.0.0",
"private": true,
"scripts": {
"test": "jest"
},
"devDependencies": {
"@testing-library/jest-dom": "latest"
}
}
```
</details>
<details>
<summary><code>test-setup.ts</code></summary>
```ts
import '@testing-library/jest-dom'
```
</details>
The app should run and all test should pass.
### ❗ Notes to reviewers
#### The problem:
1. next/jest configures jest with a transformer ([jest-transformer](https://github.com/vercel/next.js/blob/canary/packages/next/src/build/swc/jest-transformer.ts)) to compile react code with next -swc
2. the transformers configures next -swc for a given environment: Server or Client, based on jest global environment
3. Based on the environment, next -swc checks for invalid usage of `import('server-only')` `“use client”`, `export const metadata` or `export async function generateMetadata`
4. Because the global test environment is either jsdom or node, the same test suite can not include both client and server components
#### Possible mitigations
*A. Using jest projects*
When configured with [multiple projects](https://jestjs.io/docs/next/configuration/#projects-arraystring--projectconfig), Jest can launch different runners with different environment. This would allow running server-only code in node and client-only code in jsdom.
However, it requires user to completely change their jest configuration. It would also require a different setup when scaffolding new app-directory project with create-next.
*B. Using doc blocks*
Jest allows changing the environment per test file [with docBlock](https://jestjs.io/docs/configuration#testenvironment-string).
However, by the time jest is invoking next -swc on a source file to transform it, this information is gone, and next -swc is still invoked with the (wrong) global environment.
The PR #52393 provides a workaround for files with `import('server-only')`, but does not allow testing pages with metadata.
*C. Always compile for node*
Our jest-transformer could always configure next -swc for server:
- pass Server-specific validations `import('server-only')` `export const metadata` or `export async function generateMetadata`
- does not complain about `"use client"`
This is what this PR is about!
Fixes#47299
Co-authored-by: Jiachi Liu <4800338+huozhi@users.noreply.github.com>
### What?
This PR makes it easier to use Next.js with IPv6 hostnames such as `::1` and `::`.
### How?
It does so by removing rewrites from `localhost` to `127.0.0.1` introduced in #52492. It also fixes the issue where Next.js tries to fetch something like `http://::1:3000` when `--hostname` is `::1` as it is not a valid URL (browsers' `URL` class throws an error when constructed with such hosts). It also fixes `NextURL` so that it doesn't accept `http://::1:3000` but refuse `http://[::1]:3000`. It also changes `next/src/server/lib/setup-server-worker.ts` so that it uses the server's `address` method to retrieve the host instead of our provided `opts.hostname`, ensuring that no matter what `opts.hostname` is we will always get the correct one.
### Note
I've verified that `next dev`, `next start` and `node .next/standalone/server.js` work with IPv6 hostnames (such as `::` and `::1`), IPv4 hostnames (such as `127.0.0.1`, `0.0.0.0`) and `localhost` - and with any of these hostnames fetching to `localhost` also works. Server Actions and middleware have no problems as well.
This also removes `.next/standalone/server.js`'s logging as we now use `start-server`'s logging to avoid duplicates. `start-server`'s logging has also been updated to report the actual hostname.
The above pictures also demonstrate using Server Actions with Next.js after this PR.
Fixes#53171Fixes#49578
Closes NEXT-1510
Co-authored-by: Tim Neutkens <6324199+timneutkens@users.noreply.github.com>
Co-authored-by: Zack Tanner <1939140+ztanner@users.noreply.github.com>
`expect` is an unsupported header for undici: c83b084879/lib/core/request.js (L354) -- this filters it out so that the API routes don't throw an internal server error if that header is included.
Also added a test case for the previously submitted PR that was added for `content-length: 0`
- x-ref: #53843Fixes#53822 (erroneously closed by a content-length issue being fixed)
### What?
`ReactDOM.preload` is available in `react-dom@experimental` builds. If it's not available, we should fall back to `Head`+`link`
### Why?
Since `ReactDOM.preload` is only available in `react-dom@experimental` builds, certain environments (like Jest or [Storybook](https://github.com/storybookjs/storybook/issues/23661)) might have a version of `react-dom` installed that won't work with `preload()`
### How?
Closes NEXT-1482
Fixes#53272
See also: https://github.com/storybookjs/storybook/issues/23661
Co-authored-by: Steven <229881+styfle@users.noreply.github.com>
This breaks out routing handling from `next-server`, `next-dev-server`,
and `base-server` so that these are only handling the "render" work and
eventually these will be deleted completely in favor of the bundling
work being done.
The `router` process and separate `render` processes are still
maintained here although will be investigated further in follow-up to
see if we can reduce the need for these.
We are also changing the `require-cache` IPC to a single call instead of
call per entry to reduce overhead and also de-dupes handling for
starting the server between the standalone server and normal server.
To maintain support for existing turbopack route resolving this
implements the new route resolving in place of the existing
`route-resolver` until the new nextturbo API is fully landed.
After these initial changes we should continue to eliminate non-render
related code from `next-server`, `base-server`, and `next-dev-server`.
This updates our `moduleResolution` to `bundler` as this matches our heuristics much more closely so is more accurate. This shouldn't be a breaking change is it should be compatible with our previous default.
Co-authored-by: JJ Kasper <22380829+ijjk@users.noreply.github.com>
### 🧐 What's in there?
At the moment, it is not possible to test code with `import 'server-only'` in app directory.
When trying to load such file in jest (even with `testEnvironment: node`), the error will be:
```
● Test suite failed to run··
x NEXT_RSC_ERR_CLIENT_IMPORT: server-only
,-[lib/util.js:1:1]
1 | /** @jest-environment node */·
2 | import 'server-only'
: ^^^^^^^^^^^^^^^^^^^^
3 | export const PI = 3.14;
`----·
at Object.transformSync (node_modules/next/src/build/swc/index.ts:443:25)
at transformSync (node_modules/next/src/build/swc/index.ts:629:19)
at Object.process (node_modules/next/src/build/swc/jest-transformer.ts:117:25)
at ScriptTransformer.transformSource (node_modules/@jest/transform/build/ScriptTransformer.js:619:31)
at ScriptTransformer._transformAndBuildScript (node_modules/@jest/transform/build/ScriptTransformer.js:765:40)
at ScriptTransformer.transform (node_modules/@jest/transform/build/ScriptTransformer.js:822:19)·
```
In a nutshell:
- next/swc is looking for ‘server-only’ [text in the source](https://github.com/vercel/next.js/blob/canary/packages/next-swc/crates/core/src/react_server_components.rs#L576), and throw if not configured for server
- next's jest-transformer will only configure next/swc for server [if the environment is node](https://github.com/vercel/next.js/blob/canary/packages/next/src/build/swc/jest-transformer.ts#L88)
- when testing Next.js apps, your jest testEnvironment is most likely jsdom. But you can configure it [per file with docBlock](https://jestjs.io/docs/configuration#testenvironment-string), which jest-transformer ignores because it only reads the top-level configuration.
This PR fixes this, by
1. reading the docblock to configure next/swc accordingly and bypass its hardcoded guard
2. mocking `server-only` so [it does not throw](https://github.com/vercel/next.js/blob/canary/packages/next/src/compiled/server-only/index.js) when loaded (jest does not read `react-server` export from package.json)
Users would still have to annotate their `server-only` files with `/** @jest-environment node */` in order to test them.
### 🧪 How to test?
There's a full test available: `pnpm testheadless --testPathPattern jest/server-only`
Here is also a minimal reproduction:
<details>
<summary>app/layout.tsx</summary>
```typescript
export default function RootLayout({ children }: { children: React.ReactNode }) {
return (<html lang="en"><body>{children}</body></html>)
}
```
</details>
<details>
<summary>app/page.tsx</summary>
```typescript
import { PI } from '@/lib/utils'
export default function Home() {
return <h1>{PI}</h1>
}
```
</details>
<details>
<summary>lib/utils.ts</summary>
```typescript
import 'server-only'
export const PI = 3.14
```
</details>
<details>
<summary>lib/utils.test.ts</summary>
```typescript
import { PI } from './utils'
it('works', () => expect(PI).toEqual(3.14))
```
</details>
<details>
<summary>jest.config.js</summary>
```typescript
const nextJest = require('next/jest')
module.exports = nextJest({ dir: './' })({ testEnvironment: 'jsdom' })
```
</details>
### ❗ Notes to reviewers
[jest-docblock](https://packagephobia.com/result?p=jest-docblock) with dependencies is only 12.5 kB.
Fixes#47448
### What?
This PR fixes build crashing when `output: 'standalone'` and `experimental.appDir` is enabled but there is no app pages.
### How?
It does that by checking whether `.next/server/app` exists before copying the folder to `.next/standalone/...`
Closes#51828Fixes#44442Fixes#44120
As discussed in https://vercel.slack.com/archives/C03ENM5HB4K/p1687999628589119 and #51910, it makes sense to have a known list for packages (mostly polyfills) that we know are having dynamic code (`eval`, `new Function`) but are safe to run in the Edge Runtime because that dynamic code will never be executed.
Request data flow in the server
```
request ---> router worker (1) ---> ipc ---> render worker app (2)
|-----> render worker pages (3)
```
When it's hitting `_next/*` unmatched routes in standalone server, it will render 404, but when you hit `_next/*` it will render app not-found as the app router is always enabled, but router worker isn't set up with require-hook for proper built-in react version, then the app-render will fail with `./server.edge` exports not found error.
We detect if it's in the render worker, then do the app paths rendering instead of directly looking up for app paths in route worker. This could avoid unexpected accesses to the app-render
Fixes#51482Fixes#50232Closes#51506
Reverts changes in #51172
fix NEXT-1260
Small QOL improvements to `RenderResult` and `RouteModule` setup. This
also adds a test to verify that headers aren't sent on responses that
have already been sent.
---------
Co-authored-by: JJ Kasper <jj@jjsweb.site>
Fixes#50232
Passing down the private env into `'render-server'` module to pick up the proper react with require-hook in standalone server
fix NEXT-1260
This adds new `build and test` and `build and deploy` workflows in favor
of the existing massive `build, test, and deploy` workflow. Since the
new workflows will use `pull_request_target` this waits to remove the
existing workflow until the new one is tested.
While testing this new workflow flakey behavior in tests have also been
addressed. Along with the new workflow we will also be leveraging new
runners which allow us to run tests against the production binary of
`next-swc` so this avoids slight differences in tests we've seen due to
running against the dev binary.
Furthermore we will have a new flow for allowing workflow runs on PRs
from external forks which will either require a comment be checking a
box approving the run after each change or a label added by the team.
The new flow also no longer relies on `actions/cache` or similar which
have proven to be pretty unreliable.
Tests runs with the new workflow can be seen here
https://github.com/vercel/next.js/actions/runs/5100673508/jobs/9169416949
Since there is no longer a limitation that requires us to static analyze
`process.env`, this PR removes it from the build process and updates the
corresponding documentation.
If running Next.js in the custom server with both app and pages directories presented, a standalone server needs to serve all traffic and the server running in main thread should be a no-op. (cc @ijjk).
Fixes the problem described here: https://github.com/vercel/next.js/issues/49355#issuecomment-1537536063.
## What?
Removes `experimental.appDir` this was leftover from when I flipped the
switch.
Kept the config file as in the future we might add future flags and
such. It also helps that it has the types comment included so you always
get types.
<!-- Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
## For Contributors
### Improving Documentation or adding/fixing Examples
- The "examples guidelines" are followed from our contributing doc
https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md
- Make sure the linting passes by running `pnpm build && pnpm lint`. See
https://github.com/vercel/next.js/blob/canary/contributing/repository/linting.md
### Fixing a bug
- Related issues linked using `fixes #number`
- Tests added. See:
https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md
### Adding a feature
- Implements an existing feature request or RFC. Make sure the feature
request has been accepted for implementation before opening a PR. (A
discussion must be opened, see
https://github.com/vercel/next.js/discussions/new?category=ideas)
- Related issues/discussions are linked using `fixes #number`
- e2e tests added
(https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Documentation added
- Telemetry added. In case of a feature if it's used or not.
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md
## For Maintainers
- Minimal description (aim for explaining to someone not on the team to
understand the PR)
- When linking to a Slack thread, you might want to share details of the
conclusion
- Link both the Linear (Fixes NEXT-xxx) and the GitHub issues
- Add review comments if necessary to explain to the reviewer the logic
behind a change
### What?
### Why?
### How?
Closes NEXT-
Fixes #
-->
---------
Co-authored-by: JJ Kasper <jj@jjsweb.site>
