The `pnpm` version was recently updated to `8.6`, however the node version was left at `16.8`. `pnpm@8.6` requires at least `node@16.14` ([source](https://github.com/pnpm/pnpm/blob/main/pnpm/package.json#L145)).
This fixes the engines spec to avoid conflicting versions.
## Vendoring
Updates all module resolvers (node, webpack, nft for entrypoints, and nft for next-server) to consider whether vendored packages are suitable for a given resolve request and resolves them in an import semantics preserving way.
### Problem
Prior to the proposed change, vendoring has been accomplished but aliasing module requests from one specifier to a different specifier. For instance if we are using the built-in react packages for a build/runtime we might replace `require('react')` with `require('next/dist/compiled/react')`.
However this aliasing introduces a subtle bug. The React package has an export map that considers the condition `react-server` and when you require/import `'react'` the conditions should be considered and the underlying implementation of react may differ from one environment to another. In particular if you are resolving with the `react-server` condition you will be resolving the `react.shared-subset.js` implementation of React. This aliasing however breaks these semantics because it turns a bare specifier resolution of `react` with path `'.'` into a resolution with bare specifier `next` with path `'/dist/compiled/react'`. Module resolvers consider export maps of the package being imported from but in the case of `next` there is no consideration for the condition `react-server` and this resolution ends up pulling in the `index.js` implementation inside the React package by doing a simple path resolution to that package folder.
To work around this bug there is a prevalence of encoding the "right" resolution into the import itself. We for instance directly alias `react` to `next/dist/compiled/react/react.shared-subset.js` in certain cases. Other times we directly specify the runtime variant for instance `react-server-dom-webpack/server.edge` rather than `react-server-dom-wegbpack/server`, bypassing the export map altogether by selecting the runtime specific variant. However some code is meant to run in more than one runtime, for instance anything that is part of the client bundle which executes on the server during SSR and in the browser. There are workaround like using `require` conditionally or `import(...)` dynamically but these all have consequences for bundling and treeshaking and they still require careful consideration of the environment you are running in and which variant needs to load.
The result is that there is a large amount of manual pinning of aliases and additional complexity in the code and an inability to trust the package to specify the right resolution potentially causing conflicts in future versions as packages are updated.
It should be noted that aliasing is not in and of itself problematic when we are trying to implement a sort of lightweight forking based on build or runtime conditions. We have good examples of this for instance with the `next/head` package which within App Router should export a noop function. The problem is when we are trying to vendor an entire package and have the package behave semantically the same as if you had installed it yourself via node_modules
### Solution
The fix is seemingly straight forward. We need to stop aliasing these module specifiers and instead customize the resolution process to resolve from a location that will contain the desired vendored packages. We can then start simplifying our imports to use top level package resources and generally and let import conditions control the process of providing the right variant in the right context.
It should be said that vendoring is conditional. Currently we only vendor react pacakges for App Router runtimes. The implementation needs to be able to conditionally determine where a package resolves based on whether we're in an App Router context vs a Page Router one.
Additionally the implementation needs to support alternate packages such as supporting the experimental channel for React when using features that require this version.
### Implementation
The first step is to put the vendored packages inside a node_modules folder. This is essential to the correct resolving of packages by most tools that implement module resolution. For packages that are meant to be vendored, meaning whole package substitution we move the from `next/(src|dist)/compiled/...` to `next/(src|dist)/vendored/node_modules`. The purpose of this move is to clarify that vendored packages operate with a different implementation. This initial PR moves the react dependencies for App Router and `client-only` and `server-only` packages into this folder. In the future we can decide which other precompiled dependencies are best implemented as vendored packages and move them over.
It should be noted that because of our use of `JestWorker` we can get warnings for duplicate package names so we modify the vendored pacakges for react adding either `-vendored` or `-experimental-vendored` depending on which release channel the package came from. While this will require us to alter the request string for a module specifier it will still be treating the react package as the bare specifier and thus use the export map as required.
#### module resolvers
The next thing we need to do is have all systems that do module resolution implement an custom module resolution step. There are five different resolvers that need to be considered
##### node runtime
Updated the require-hook to resolve from the vendored directory without rewriting the request string to alter which package is identified in the bare specifier. For react packages we only do this vendoring if the `process.env.__NEXT_PRIVATE_PREBUNDLED_REACT` envvar is set indicating the runtime is server App Router builds. If we need a single node runtime to be able to conditionally resolve to both vendored and non vendored versions we will need to combine this with aliasing and encode whether the request is for the vendored version in the request string. Our current architecture does not require this though so we will just rely on the envvar for now
##### webpack runtime
Removed all aliases configured for react packages. Rely on the node-runtime to properly alias external react dependencies. Add a resolver plugin `NextAppResolverPlugin` to preempt perform resolution from the context of the vendored directory when encountering a vendored eligible package.
##### turbopack runtime
updated the aliasing rules for react packages to resolve from the vendored directory when in an App Router context. This implementation is all essentially config b/c the capability of doing the resolve from any position (i.e. the vendored directory) already exists
##### nft entrypoints runtime
track chunks to trace for App Router separate from Pages Router. For the trace for App Router chunks use a custom resolve hook in nft which performs the resolution from the vendored directory when appropriate.
##### nft next-server runtime
The current implementation for next-server traces both node_modules and vendored version of packages so all versions are included. This is necessary because the next server can run in either context (App vs Page router) and may depend on any possible variants. We could in theory make two traces rather than a combined one but this will require additional downstream changes so for now it is the most conservative thing to do and is correct
Once we have the correct resolution semantics for all resolvers we can start to remove instances targeting our precompiled instances for instance making `import ... from "next/dist/compiled/react-server-dom-webpack/client"` and replacing with `import ... from "react-server-dom-webpack/client"`
We can also stop requiring runtime specific variants like `import ... from "react-server-dom-webpack/client.edge"` replacing it with the generic export `"react-server-dom-webpack/client"`
There are still two special case aliases related to react
1. In profiling mode (browser only) we rewrite `react-dom` to `react-dom/profiling` and `scheduler/tracing` to `scheduler/tracing-profiling`. This can be moved to using export maps and conditions once react publishses updates that implement this on the package side.
2. When resolving `react-dom` on the server we rewrite this to `react-dom/server-rendering-stub`. This is to avoid loading the entire react-dom client bundle on the server when most of it goes unused. In the next major react will update this top level export to only contain the parts that are usable in any runtime and this alias can be dropped entirely
There are two non-react packages currently be vendored that I have maintained but think we ought to discuss the validity of vendoring. The `client-only` and `server-only` packages are vendored so you can use them without having to remember to install them into your project. This is convenient but does perhaps become surprising if you don't realize what is happening. We should consider not doing this but we can make that decision in another discussion/PR.
#### Webpack Layers
One of the things our webpack config implements for App Router is layers which allow us to have separate instances of packages for the server components graph and the client (ssr) graph. The way we were managing layer selection was a but arbitrary so in addition to the other webpack changes the way you cause a file to always end up in a specific layer is to end it with `.serverlayer`, `.clientlayer` or `.sharedlayer`. These act as layer portals so something in the server layer can import `foo.clientlayer` and that module will in fact be bundled in the client layer.
#### Packaging Changes
Most package managers are fine with this resolution redirect however yarn berry (yarn 2+ with PnP) will not resolve packages that are not defined in a package.json as a dependency. This was not a problem with the prior strategy because it was never resolving these vendored packages it was always resolving the next package and then just pointing to a file within it that happened to be from react or a related package.
To get around this issue vendored packages are both committed in src and packed as a tgz file. Then in the next package.json we define these vendored packages as `optionalDependency` pointing to these tarballs. For yarn PnP these packed versions will get used and resolved rather than the locally commited src files. For other package managers the optional dependencies may or may not get installed but the resolution will still resolve to the checked in src files. This isn't a particularly satisfying implemenation and if pnpm were to be updated to have consistent behavior installing from tarballs we could actually move the vendoring entirely to dependencies and simplify our resolvers a fair bit. But this will require an upstream chagne in pnpm and would take time to propogate in the community since many use older versions
#### Upstream Changes
As part of this work I landed some other changes upstream that were necessary. One was to make our packing use `npm` to match our publishing step. This also allows us to pack `node_modules` folders which is normally not supported but is possible if you define the folder in the package.json's files property.
See: #52563
Additionally nft did not provide a way to use the internal resolver if you were going to use the resolve hook so that is now exposed
See: https://github.com/vercel/nft/pull/354
#### additional PR trivia
* When we prepare to make an isolated next install for integration tests we exclude node_modules by default so we have a special case to allow `/vendored/node_modules`
* The webpack module rules were refactored to be a little easier to reason about and while they do work as is it would be better for some of them to be wrapped in a `oneOf` rule however there is a bug in our css loader implementation that causes these oneOf rules to get deleted. We should fix this up in a followup to make the rules a little more robuts.
## Edits
* I removed `.sharedlayer` since this concept is leaky (not really related to the client/server boundary split) and it is getting refactored anyway soon into a precompiled runtime.
Spinning out from #37151 and my draft PR #52845, this enables the two
basic recommended rulesets from
[typescript-eslint](https://typescript-eslint.io) for the Next.js
monorepo source code:
*
[`plugin:@typescript-eslint/recommended`](https://typescript-eslint.io/linting/configs#recommended):
Our base recommended rules that detect common bugs or _(non-stylistic)_
TypeScript bad practices
*
[`plugin:@typescript-eslint/stylistic`](https://typescript-eslint.io/linting/configs#stylistic):
Our base starting stylistic recommended for keeping codebases visually
consistent and avoiding out-of-practice visual constructs
The process I used is pretty standard (see
https://github.com/typescript-eslint/typescript-eslint/issues/6760 for
other repos it was done on):
1. Enable those base recommended presets
2. Remove any rule settings that are now redundant
3. Reconfigure any rule whose default settings didn't seem to make sense
for this codebase
4. Add a `// Todo: ...` comment, and under it, add a disable for any
rule that immediately reported a lot of complaints
Note that this only enables the presets internally. It doesn't impact
what end-users of published packages such as Next.js or
`create-next-app` experience. That's a separate task in #52845.
I also didn't fix any existing warning from the `canary` branch. Would
you like me to do that? My preference would be a separate PR to get it
in more quickly.
Any code changes are commented inline.
---------
Co-authored-by: Steven <steven@ceriously.com>
This refactors the changes from my previous PR to allow smooth scrolling for the appDir case -- `componentDidUpdate` isn't a reliable way to check if only the hash has changed. This adds a property to `focusAndScrollRef` and compares canonicalUrls (sans hash fragment)
- Original https://github.com/vercel/next.js/pull/52915
Co-authored-by: Tim Neutkens <6324199+timneutkens@users.noreply.github.com>
You'll probably want to disable whitespace in the diff
## Description
This allows for better editor support by using `describe` or functions called `describe` with the same syntax instead of custom names.
Changes:
- `nextTestSetup` can be used instead of `createNextDescribe` keeping the same behaviour but being called inside a `describe` "block" (not applied everywhere)
- `getSnapshotTestDescribe` replaced with a custom `describe.each`
- `sandbox` helper function for `acceptance`/`acceptance-app` merged into a single shared one
- `outdent` to remove the indent from inline files in tests which helps with consistent snapshots
Reverts vercel/next.js#51406
This pnpm bump seems to be causing lockfile issues with the react
versions which causes context issues since we need to share the same
react version in the monorepo.
### What?
Fixespnpm/pnpm#6424
### Why?
So users of Node 20 can contribute to Next.js
### How?
Upgrades pnpm with back compatibility to patch `ERR_INVALID_THIS` error.
If Node 14 support isn't necessary, I'd recommend upgrading to pnpm 8
for continued feature support and committing the lockfile.
---------
Co-authored-by: JJ Kasper <jj@jjsweb.site>
### What?
WEB-1150.
This PR is an attempt to upload next.js's test results into datadog test trace to track its status.
Originally it tried to use automatic trace injection (dd-trace/ci/init). However, due to some of custom environments / setup we use it was not compatible out of the box. Instead, this PR injects a new test reportert to generate junit report, then upload it at once at the end of the testing pipeline.
The reporter is configured to run when necessary variables set, local run should not be affected.
One thing to note is this report will not count retry results, as it'll create duplicated test entry with multiple results since we runts jest per individual test. This'll allow to detect flaky test easier, but also it means we'll get bit of skewed test results compare to the real world as first failure will be accounted as test fail immediately.
There are some incoming docs / MDX changes where prettier will throw an error when using the older version. Updating prettier before I bring in those changes.
Looks like the most notable change is adding parentheses around `typeof` checks in TypeScript.
**Before**
```
export type Locale = typeof i18n['locales'][number]
```
**After**
```
export type Locale = (typeof i18n)['locales'][number]
```
To resolve issue #49382, we found layer doesn't get applied for dynamic imports, so we fixed it on webpack side in https://github.com/webpack/webpack/pull/17310
This PR is to upgrade webpack to 5.86.0 with that patch as a precedence. After this we need to fix the `next/image` client components is missing in client reference manifest when using dynamic imports to fix the issue.
For RSC server layer so far we bundle all dependencies, ESM format is the better one rather than CJS to analyze and tree-shake out the unused parts. This PR changes pick the condition names that are in ESM format first for server layer.
Also fixes the misorder of condition names of edge runtime, `conditionNames` should only contain either ESM or CJS, previously the main fields are mixed with conditon names which is not expected for webpack, we separate them now.
Since we're picking ESM instead CJS now, the error of require `exports * from` doesn't exist anymore, but if you're using a CJS dependency which require a ESM package, it will error. This is the existing behavior for our webpack configuration but could happen on server layer bundling
Other related changes:
* Imports are hoisted in ESM, so migrate`enhanceGlobals` to a imported module
* Use `...` to pick the proper imports by import expression, and prefer the `react-server` / `edge-light` condition names for corresponding cases
* Remove edge SSR duplicated `middleware` export checking
### What?
Another attempt to https://github.com/vercel/next.js/pull/50619 and WEB-1150, trying to apply setup guard more throughly.
I still do not know why original PR passed CI but fails to subsequent PRs after merge, but hope this could be a right guard to prevent unexpected failures.
Hello,
We removed some core non-necessary dependencies that make Edge Runtime
smaller 🙂.
Also, Edge Runtime is exposing `WebSocket`, so nothing additional should
be done.
Closes https://github.com/vercel/next.js/issues/50760
---------
Co-authored-by: Wyatt Johnson <accounts+github@wyattjoh.ca>
Bumping `@edge-runtime/*` <picture data-single-emoji=":edge-runtime:" title=":edge-runtime:"><img class="emoji" width="20" height="auto" src="https://emoji.slack-edge.com/T0CAQ00TU/edge-runtime/b940e917443aa49f.png" alt=":edge-runtime:" align="absmiddle"></picture> packages to their latest beta, as we had some huge improvements and big changes on how it works internally.
* Using the latest `undici` which provides a WebSocket implementation
* Less polyfill usage as we load modules in the Node.js realm, and can reuse modules from Node.js as time goes on <picture data-single-emoji=":just-right2:" title=":just-right2:"><img class="emoji" width="20" height="auto" src="https://emoji.slack-edge.com/T0CAQ00TU/just-right2/588cf34d02f4b3bd.png" alt=":just-right2:" align="absmiddle"></picture>
* `instanceof` checks now work within the realm <picture data-single-emoji=":mind_blown:" title=":mind_blown:"><img class="emoji" width="20" height="auto" src="https://emoji.slack-edge.com/T0CAQ00TU/mind_blown/0186b6f181040126.gif" alt=":mind_blown:" align="absmiddle"></picture>
This adds new `build and test` and `build and deploy` workflows in favor
of the existing massive `build, test, and deploy` workflow. Since the
new workflows will use `pull_request_target` this waits to remove the
existing workflow until the new one is tested.
While testing this new workflow flakey behavior in tests have also been
addressed. Along with the new workflow we will also be leveraging new
runners which allow us to run tests against the production binary of
`next-swc` so this avoids slight differences in tests we've seen due to
running against the dev binary.
Furthermore we will have a new flow for allowing workflow runs on PRs
from external forks which will either require a comment be checking a
box approving the run after each change or a label added by the team.
The new flow also no longer relies on `actions/cache` or similar which
have proven to be pretty unreliable.
Tests runs with the new workflow can be seen here
https://github.com/vercel/next.js/actions/runs/5100673508/jobs/9169416949
This PR does three things:
- Vendors the package `react-server-dom-webpack@experimental` as
`react-server-dom-webpack-experimental` similar to React and React DOM
- Upgrades all vendored React packages
- Re-lands the `experimentalReact` flag to switch between `@next` and
`@experimental` channels of React for app dir.
Fix NEXT-926.
### What?
Exposing `globalThis.crypto`, based on [Node.js' WebCrypto
API](https://nodejs.org/api/globals.html#crypto_1)
### Why?
Similar to `fetch`, `crypto` is a popular API that is currently not
available on `globalThis` in all active Node.js versions yet.
This can help library authors to create runtime-agnostic packages.
### How?
Node.js already has the WebCrypto API that can be imported, we just
expose it on `globalThis` in Node.js versions where this is not
available.
Closes NEXT-1063
[Slack
thread](https://vercel.slack.com/archives/C03KAR5DCKC/p1681821510191059)
