Closes WEB-1518
### What?
This PR refactors test cases from next-dev, tests basic tailwind rendering with turbopack. It still runs against plain next.js too. While checking, noticed there's a stub fixture (css-fixture/with-tailwindcss) but those fixture seems not being used in the test. Moved those and utilized. Since this test runs against next.js / turbopack both anyway, moving fixture should not be a huge regression I believe.
### What?
Switch the default for `--turbo` to the new `--experimental-turbo`, remove the old code in next.js
### Why?
The new approach will be used in future
Closes WEB-1506
While investigating the HMR event types I noticed `pong` is not being used in Pages Router nor in App Router.
This PR removes the code that sends it as it's essentially dead code.
This fix ensures the case that if you `import { foo } from './index'`,
where:
index.js:
```js
export * from './subpkg'
```
subpkg.js:
```js
const unrelatedExpressions = ...
export { foo } from './foo'
export { bar } from './bar'
```
Previously we'll transform the proxy to the second module to `export {
foo } from './subpkg'`. With this fix it will be correctly optimized as
`export { foo } from './foo'` which is a shorter path.
### What
When you only have `pages/` folder and custom `_app.js`, editing the
`_app.js` file when unexpectedly lead to full page reload. This is
because the `pages/_app` chunk is included into `pages/_document` chunk
as they all pass through the `next-route-loader` which forms the router
module and other userland modules including `/_app` and `/_document`.
That leads to a change: whenever you modify `_app.js`, chunk hash of
`/_document` will always be updated as it's built as a new chunk.
Found this difference by looking at the document chunk with nextjs in
early 13.4.x releases.
### How
We'll check at the entries creation point to filter out `/_app` and
`/_document` so that we won't have them both bundled for `_document`
chunk. Then when we won't trigger the hmr path to reload the page since
_document chunk hash is updated.
Closes NEXT-1568
Fixes#53614
## Initial Pass
The initial pass applies to all user code. If an import (`foo`) matches one of the packages from our list, say:
```js
// user code
import { a } from 'foo'
import { otherThings } from './code'
```
The initial pass transforms it into:
```js
// user code
import { a } from '__barrel_optimize__?names=a!=!foo'
import { otherThings } from './code'
```
## Barrel Optimizations
This `__barrel_optimize__` loader is used to optimize the imports of "barrel" files that have many
re-exports. Currently, both Node.js and Webpack have to enter all of these
submodules even if we only need a few of them.
For example, say a file `foo.js` with the following contents:
```js
export { a } from './a'
export { b } from './b'
export { c } from './c'
...
```
If the user imports `a` only, this loader will accept the `names` option to
be `['a']`. Then, it request the `__barrel_transform__` SWC loader to load
`foo.js` (via `this.loadModule` Webpack API) and receive the following info:
```js
export const __next_private_export_map__ = '[["./a","a","a"],["./b","b","b"],["./c","c","c"],...]'
```
The export map, generated by SWC, is a JSON that represents the exports of
that module, their original file, and their original name (since you can do
`export { a as b }`).
Then, this loader can safely remove all the exports that are not needed and
re-export the ones from `names`:
```js
export { a } from './a'
```
That's the basic situation and also the happy path.
## Wildcard Exports
For wildcard exports (e.g. `export * from './a'`), it becomes a bit more complicated.
Say `foo.js` with the following contents:
```js
export * from './a'
export * from './b'
export * from './c'
...
```
If the user imports `bar` from it, SWC can never know which files are going to be
exporting `bar`. So, we have to keep all the wildcard exports and do the same
process recursively. This loader will return the following output:
```js
export * from '__barrel_optimize__?names=bar&wildcard!=!./a'
export * from '__barrel_optimize__?names=bar&wildcard!=!./b'
export * from '__barrel_optimize__?names=bar&wildcard!=!./c'
...
```
The "!=!" tells Webpack to use the same loader to process './a', './b', and './c'.
After the recursive process, the "inner loaders" will either return an empty string
or:
```js
export * from './target'
```
Where `target` is the file that exports `bar`.
## Non-Barrel Files
If the file is not a barrel, we can't apply any optimizations. That's because
we can't easily remove things from the file. For example, say `foo.js` with:
```js
const v = 1
export function b () {
return v
}
```
If the user imports `b` only, we can't remove the `const v = 1` even though
the file is side-effect free. In these caes, this loader will simply re-export
`foo.js`:
```js
export * from './foo'
```
Besides these cases, this loader also carefully handles the module cache so
SWC won't analyze the same file twice, and no instance of the same file will
be accidentally created as different instances.
---
- [x] Disable this loader for Jest
- [x] Add tests for SWC loader caching
- [x] Add tests for external modules
- [x] Add tests for pages
- [x] Add tests for recursive `export *`s
---
Closes#54038, closes#54286.
A continuation of #51864 with RedirectType used in an e2e test and some
ts-ignores removed from the codebase. Happy to split it into two PRs but
I left comments in the two consequential files
---------
Adding a default app router not-found error page in production. We introduced a custom global not-found page if you have `not-found.js` file under app directory. Next.js will still routed you to default pages 404 if there's no routes matched when you don't have a custom `not-found.js`.
This PR creates a default layout (only having `html` and `body`) and will use the default not-found component as children to build a default 404 page for app router, and server will route to that page when there's any app router pages existed. This will also fix the hmr issue that when you hit a pathname doesn't exist, nextjs will compile `/_error` for you which is not expected, now we're using `/not-found` instead
Closes NEXT-1359
## Implementation
Base on #54530, we're implementing a `optimize_barrel` transform to
optimize barrel files to only export the names we need. If the
transformed file isn't a "barrel file", we just re-export the names from
it without any transformation.
Take `lucide-react` as an example, with #54530 we are able to transform
```js
import { IceCream } from 'lucide-react'
```
to
```js
import { IceCream } from '__barrel_optimize__?names=IceCream!=!lucide-react?__barrel_optimize_noop__=IceCream'
```
And then, we apply that new request with a new Webpack module rule to
use the SWC loader with option `optimizeBarrelExports: ['IceCream']`,
which eventually got passed to this new `optimize_barrel` transform and
does the optimization.
## Notes
We'll have to add a new `getModularizeImportAliases` alias list to map
`lucide-react` to the ESM version, as we have the `['main', 'module']`
resolve order for the server compiler. Otherwise this optimization
doesn't work in that compiler.
There's no e2e test added because it's already covered by the
`modularize-imports` test as we removed the default `lucide-react`
transform rules and it still works.
We'll need to test other libs before migrating them to the new
`optimizePackageImports` option.
---
Closes#54571, closes#53605, closes#53789, closes#53894, closes
#54063.
### What?
* add support for middleware manifest and edge adapter for pages API
routes
* improve the error reporting a tiny bit
### Why?
### How?
Closes WEB-1428
This is the first step to enable the automatic "modularize imports"
optimization for some libraries. It transforms named imports like
`import { A, B, C as F } from 'foo'` to a special loader string: `import
{ A, B, C as F } from "barrel-optimize-loader?names=A,B,C!foo"`.
In a follow-up PR we'll apply corresponding optimization with another
SWC transformer.
This removes the client-side polling logic that we're doing on every HMR
tick in favor of events like `addedPage` / `removedPage`. It was leading
to a lot of misc issues & confusion surrounding error/404 pages being
polled needlessly.
Instead this updates `addedPage` / `removedPage` to be emitted on the
server, so that we can leverage those hooks to reload when necessary.
Fixes#10024Fixes#51132
Closes NEXT-1372
related discussion: https://github.com/vercel/next.js/discussions/40000
---------
## What
Adds a reworked version of webpack's [built-in memory cache garbage
collection
plugin](853bfda35a/lib/cache/MemoryWithGcCachePlugin.js (L15))
that is more aggressive in cleaning up unused modules than the default.
The default marks 1/5th of the modules as "up for potentially being
garbage collected". The new plugin always checks all modules. In my
testing this does not cause much overhead compared to the current
approach which leverages writing to two separate maps. The change also
makes the memory cache eviction more predictable: when an item has not
been accessed for 5 compilations it is evicted from the memory cache, it
could still be in the disk cache.
In order to test this change I had to spin up the benchmarks but these
were a bit outdated so I've cleaned up the benchmark applications.
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
### What?
Let Next.js enable React Strict Mode by default for the App Router.
This caught a bug with the `AppRouterAnnouncer` added in #47018 which
should be fixed as well.
### Why?
This logic:
a0d1d728b9/packages/next/src/build/webpack-config.ts (L326-L333)
assumes that the default value of `reactStrictMode` is `null`, but it
was set to `false` in #46656.
### How?
This PR reverts the default value to `null`.
Closes NEXT-1484
Fixes#53315
---------
Co-authored-by: Tim Neutkens <tim@timneutkens.nl>
This PR merges the app renderer worker into the router process. This
improves the memory overhead mostly.
There're future work to do to get rid of the IPC server for router and
app renderer, as they're now merged in one process.
Fixes NEXT-1492
### What?
Fixing outdated import mapping with the latest `lucide-react` changes. See [lucide-icons/lucide@v0.264.0](https://github.com/lucide-icons/lucide/releases/tag/v0.264.0)
Closes: lucide-icons/lucide#1482
### Why?
Import mapping was outdated
### How?
By updating the default config and the tests.
## Notes
Maintainer here from @lucide-icons.
To this day I didn't know the Nextjs default config file has predefined hardcoded modularize imports.
I was a bit surprised that there is a default mapping for libraries that could change in the future like Lucide.
I'm not in favor of this hardcoded "modularize imports", because this creates a hard dependency on libraries. With Lucide, we still have not reached 1.0 and will sometimes rename icons and add aliases for them. So issues will occur in the future, and constantly updating this config doesn't feel right.
Is there, not a better way? Like for example, a package exposing a config file containing the import mapping?
This way import map will automatically be in sync with changes in a package.
Let me know what you think of this.
In the current version of Next.js there are 4 processes when running in
production:
- Server
- Routing
- Rendering Pages Router
- Rendering App Router
This setup was introduced in order to allow App Router and Pages Router
to use different versions of React (i.e. Server Actions currently
requires react@experimental to function). I wrote down more on why these
processes exist in this comment:
https://github.com/vercel/next.js/issues/49929#issuecomment-1637185156
This PR combines the Server and Routing process into one handler, as the
"Server" process was only proxying to the Routing process. In my testing
this caused about ~2x the amount of memory per request as the response
body was duplicated between the processes. This was especially visible
in the case of that memory leak in Node.js 18.16 as it grew memory usage
on both sides quickly.
In the process of going through these changes I found a couple of other
bugs like the propagation of values to the worker processes not being
awaited
([link](https://github.com/vercel/next.js/pull/53523/files#diff-0ef09f360141930bb03263b378d37d71ad9432ac851679aeabc577923536df84R54))
and the dot syntax for propagation was not functioning.
It also seemed there were a few cases where watchpack was used that
would cause many more files to be watched than expected, for now I've
removed those cases, specifically the "delete dir while running" and
instrument.js hmr (instrument.js is experimental). Those tests have been
skipped for now until we can revisit them to verfiy it
I've also cleaned up the types a bit while I was looking into these
changes.
### Improvement
⚠️ Important preface to this, measuring memory usage / peak usage is not
super reliable especially when JS gets garbage collected. These numbers
are just to show the rough percentage of less memory usage.
#### Baseline
Old:
```
next-server: 44.8MB
next-router-worker: 57.5MB
next-render-worker-app: 39,6MB
next-render-worker-pages: 39,1MB
```
New:
```
next-server: Removed
next-router-worker: 64.4MB
next-render-worker-app: 43.1MB (Note: no changes here, this shows what I meant by rough numbers)
next-render-worker-pages: 42.4MB (Note: no changes here, this shows what I meant by rough numbers)
```
Overall win: ~40MB (process is removed)
#### Peak usage
Old:
```
next-server: 118.6MB
next-router-worker: 223.7MB
next-render-worker-app: 32.8MB (I ran the test on a pages application in this case)
next-render-worker-pages: 101.1MB
```
New:
```
next-server: Removed
next-router-worker: 179.1MB
next-render-worker-app: 33.4MB
next-render-worker-pages: 117.5MB
```
Overall win: ~100MB (but it scales with requests so it was about ~50% of
next-router-worker constantly)
Related: #53523
---------
Co-authored-by: JJ Kasper <jj@jjsweb.site>
When using imports from `next/headers` in a layout or page,
`StaticGenerationBailout` will throw an error to indicate Next.js should
fallback to dynamic rendering. However, when async context is lost, this
error is uncaught and leads to a confusing error message at build time.
This attempts to improve DX surrounding this error by linking out to a
page that explains when it might happen. I've also tweaked
`StaticGenerationBailout` to always throw a fully descriptive reason as
opposed to just `DynamicServerError: Dynamic server usage: cookies`
Closes NEXT-1181
Fixes#49373
---------
Co-authored-by: Lee Robinson <me@leerob.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
### What?
This PR makes it so calling `experimental_useOptimistic` throws an error telling you it only works in a client component. Because the Next docs have an example of renaming it into `useOptimistic` in the import, I also added that as a forbidden import. There may be a better way to do this, if so, please let me know.
Fixes#53312
### Why?
Currently, the error you get says `(0 , react__WEBPACK_IMPORTED_MODULE_1__.experimental_useOptimistic) is not a function or its return value is not iterable`. This is misleading.
<img width="1043" alt="Screenshot 2023-07-28 at 3 30 10 PM" src="https://github.com/vercel/next.js/assets/12662580/ee16fd84-633d-47a1-8db4-cfc050546614">
### How?
Adds `experimental_useOptimistic` to the lists of forbidden imports. Adds some specific tests around this, but I'm not sure they're necessary, looking at how the other tests are written.
Co-authored-by: Zack Tanner <1939140+ztanner@users.noreply.github.com>
### What?
* fixes problems in Next.rs API introduced by #52846
* adds test infrastructure for experimental turbo testing
* adds two test cases to verify the infrastructure
* add grouping of output logs in run-tests
* simplify template loading
### Why?
### How?
This PR adds some improved `modularizeImports` rules to ensure that some
of the popular UI libraries (mostly icons) can be correctly optimized
instead of creating thousands of modules. Here's an example of applying
this to `lucide-react`:
With https://github.com/swc-project/plugins/pull/196 being landed, we
can add different transform rules for specific names in the import. For
example, for `lucide-react` there're now 3 transforms:
- `'Lucide(.*)': ...`
- `'(.*)Icon': ...`
- `'*': ...`
On top of that, another critical change made in this PR is the
introducing of `modularize-import-loader`. With this new loader, we can
delegate the task of re-exporting the value to Webpack, where we can
adjust the type of the export and the target path. This is very
necessary and can't be done in SWC alone because libs like
`lucide-react` is using `.mjs` and don't have `exports` values for the
individual icons in the package.json files.
Because of that, if we simply transform the expression to `import Icon
from 'lucide-react/esm/icons/foo` it will fail because:
1. It's missing a `.mjs` extension
2. It doesn't have `/esm/icons/foo.mjs` in package.json `exports`
For 1), in fact that they moved to `.mjs` only in a recent upgrade so we
can't even hard code it to be `.mjs` in the transform (breaks old
versions).
Because of all the above, I decided to go with Webpack's own resolution
logic with the loader as a _temporary_ solution. It's temporary because
it's still assuming that the file structure of these libs doesn't
change.
### What?
- Visiting a page in the app router without a proper component export doesn't show the dev overlay, but logs errors to the console
- When it does show the error overlay (e.g. during an HMR event), the error message was sharing the module code itself rather than the component path, making it hard to debug
### Why?
`createComponentTree` can throw these errors before the AppRouter tree is mounted, leaving the errors uncaught by the dev overlay.
### How?
This wraps the server root in the `ReactDevOverlay` when in dev mode with a minimal "HMR" for when the server component is edited (to reload the page).
Closes NEXT-308
This updates our `moduleResolution` to `bundler` as this matches our heuristics much more closely so is more accurate. This shouldn't be a breaking change is it should be compatible with our previous default.
Co-authored-by: JJ Kasper <22380829+ijjk@users.noreply.github.com>
- Adds `next-router-worker` as the process title for the routing worker.
- Adds `next-render-worker-app` as the process title for the rendering
worker for `app`
- Adds `next-render-worker-pages` as the process title for the rendering
worker for `pages`
Currently it's hard to find these processes in e.g. activity monitor,
this helps explain what they are used for when someone is looking at the
running processes.
<!-- Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
## For Contributors
### Improving Documentation
- Run `pnpm prettier-fix` to fix formatting issues before opening the
PR.
- Read the Docs Contribution Guide to ensure your contribution follows
the docs guidelines:
https://nextjs.org/docs/community/contribution-guide
### Adding or Updating Examples
- The "examples guidelines" are followed from our contributing doc
https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md
- Make sure the linting passes by running `pnpm build && pnpm lint`. See
https://github.com/vercel/next.js/blob/canary/contributing/repository/linting.md
### Fixing a bug
- Related issues linked using `fixes #number`
- Tests added. See:
https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md
### Adding a feature
- Implements an existing feature request or RFC. Make sure the feature
request has been accepted for implementation before opening a PR. (A
discussion must be opened, see
https://github.com/vercel/next.js/discussions/new?category=ideas)
- Related issues/discussions are linked using `fixes #number`
- e2e tests added
(https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Documentation added
- Telemetry added. In case of a feature if it's used or not.
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md
## For Maintainers
- Minimal description (aim for explaining to someone not on the team to
understand the PR)
- When linking to a Slack thread, you might want to share details of the
conclusion
- Link both the Linear (Fixes NEXT-xxx) and the GitHub issues
- Add review comments if necessary to explain to the reviewer the logic
behind a change
### What?
### Why?
### How?
Closes NEXT-
Fixes #
-->
### Issue
When you edit .env* files, the pages under app dir that using env vars are not triggering hot reload
### Fix
Triggering serverComponentChanges hot reload action when we detect env or tsconfig related change. There's a time period that we need to wait before the compilation is finished. First we save a flag `reloadOnDone` if we need to reload when after compilation is done, by determining if `envChange` is `true` (we already know this in dev server). Then in the compilation hooks, we refresh RSC page once it's finished.
### Extra change
since we're checking `event.action` in client hot reloader, and throwing error for unknown action, filter devPagesManifestUpdate out from unexpected action as it sometimes triggered as error in console. Introduced in #51516
Fixes NEXT-1261
You'll probably want to disable whitespace in the diff
## Description
This allows for better editor support by using `describe` or functions called `describe` with the same syntax instead of custom names.
Changes:
- `nextTestSetup` can be used instead of `createNextDescribe` keeping the same behaviour but being called inside a `describe` "block" (not applied everywhere)
- `getSnapshotTestDescribe` replaced with a custom `describe.each`
- `sandbox` helper function for `acceptance`/`acceptance-app` merged into a single shared one
- `outdent` to remove the indent from inline files in tests which helps with consistent snapshots
When working with React Server Components, it's essential to have a
clear understanding of how they function and how to troubleshoot any
errors that may arise. This PR adds links to our documentation whenever
an error related to React Server Components occurs.
This breaks up some of our longest running tests which allows more
parallelizing of the tests. This also moves turbopack tests filtering
back to an allow list as it is running a lot of unrelated tests
currently which isn't ideal. We should only be running against tests
that are explicitly testing turbopack e.g. build tests should not be
duplicated in the turbopack group.
```sh
test/integration/css/test/group-1.test.js: 762.655
test/integration/edge-runtime-module-errors/test/index.test.js: 695.309
test/integration/css/test/group-2.test.js: 671.848
test/integration/i18n-support/test/index.test.js: 518.173
test/integration/scss-modules/test/index.test.js: 451.704
test/integration/css-features/test/index.test.js: 417.318
test/integration/css-modules/test/index.test.js: 403.405
test/integration/eslint/test/index.test.js: 381.563
test/integration/500-page/test/index.test.js: 371.134
test/integration/telemetry/test/index.test.js: 367.691
test/development/acceptance-app/ReactRefreshLogBox.test.ts: 335.878
test/integration/create-next-app/templates.test.ts: 334.01
test/integration/scss/test/group-2.test.js: 327.255
test/integration/scss/test/group-1.test.js: 318.574
test/integration/edge-runtime-configurable-guards/test/index.test.js: 313.834
test/e2e/instrumentation-hook/instrumentation-hook.test.ts: 294.618
test/development/acceptance-app/error-recovery.test.ts: 283.355
test/e2e/app-dir/app/vercel-speed-insights.test.ts: 278.242
test/integration/create-next-app/index.test.ts: 272.442
```
This adds new `build and test` and `build and deploy` workflows in favor
of the existing massive `build, test, and deploy` workflow. Since the
new workflows will use `pull_request_target` this waits to remove the
existing workflow until the new one is tested.
While testing this new workflow flakey behavior in tests have also been
addressed. Along with the new workflow we will also be leveraging new
runners which allow us to run tests against the production binary of
`next-swc` so this avoids slight differences in tests we've seen due to
running against the dev binary.
Furthermore we will have a new flow for allowing workflow runs on PRs
from external forks which will either require a comment be checking a
box approving the run after each change or a label added by the team.
The new flow also no longer relies on `actions/cache` or similar which
have proven to be pretty unreliable.
Tests runs with the new workflow can be seen here
https://github.com/vercel/next.js/actions/runs/5100673508/jobs/9169416949
### What?
create internal modules via `context.process` with
`ReferenceType::Internal`
### Why?
We need this for future refactoring where we want to the internal
modules with transitions.
### Turbopack Changes
* https://github.com/vercel/turbo/pull/5095 <!-- Tobias Koppers - allow
to create internal modules via AssetContext -->
* https://github.com/vercel/turbo/pull/5104 <!-- Alex Kirszenberg -
Stable chunk list ident -->
* https://github.com/vercel/turbo/pull/5106 <!-- Tobias Koppers -
followup fix -->
---------
Co-authored-by: Alex Kirszenberg <alex.kirszenberg@vercel.com>
This exports the named exports from the user's page route file and
exports a new `routeModule` stub. This stub will be the new rendering
mechanism used for pages within `pages/` and will serve to allow us to
deprecate large chunks of the server code to instead live within the
pages entrypoints.
TLDR;
- Route Modules now accept a `definition` so that they have their full
route definition
- Exports are now hoisted from the userland module by name to prevent
name collisions
- A new `PagesRouteModule` stub was added that will in the future
contain all the pages rendering logic
---------
Co-authored-by: Tim Neutkens <tim@timneutkens.nl>
