This updates to handle detecting `getStaticProps`/`getServerSideProps` correctly during build when `experimental-edge` is being used. This also fixes not parsing dynamic route params correctly with the edge runtime and sets up the handling needed for the static generation for app opened in the below mentioned PR.
## Bug
- [x] Related issues linked using `fixes #number`
- [x] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
Fixes: [slack thread](https://vercel.slack.com/archives/C0289CGVAR2/p1661554455121189)
x-ref: https://github.com/vercel/next.js/pull/39884
When a user tried to have the following Edge Function:
```ts
export default () => fetch("https://example.vercel.sh");
```
The Edge Function were failing.
Why is that?
When `fetch` was called, an implicit `Accept-Encoding` header was added
to allow the origin to return a compressed response. Then, the origin
will set the `Content-Encoding` header in the response, to let the
client know that the body needs to be decompressed in order to be read.
That creates an issue though: `response.body` will be a
`ReadableStream<Uint8Array>`, or, a stream that contains binary data
that decodes into _the uncompressed data_ (or, plain text!).
What it means, is that `response.body` is uncompressed data, while
`response.headers.get('content-encoding')` is marking the response body
as compressed payload. This confuses the HTTP clients and makes them fail.
This commit removes the `content-encoding`, `transfer-encoding` and
`content-length` headers from the response, as the Next.js server _always_
streams Edge Function responses.
## Bug
- [ ] Related issues linked using `fixes #number`
- [x] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
Follow-up to the earlier enabling of classes/variables etc.
Bug
Related issues linked using fixes #number
Integration tests added
Errors have helpful link attached, see contributing.md
Feature
Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
Related issues linked using fixes #number
Integration tests added
Documentation added
Telemetry added. In case of a feature if it's used or not.
Errors have helpful link attached, see contributing.md
Documentation / Examples
Make sure the linting passes by running pnpm lint
The examples guidelines are followed from our contributing doc
Co-authored-by: Steven <steven@ceriously.com>
WIP.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm lint`
- [ ] The examples guidelines are followed from [our contributing doc](https://github.com/vercel/next.js/blob/canary/contributing.md#adding-examples)
Co-authored-by: Shu Ding <3676859+shuding@users.noreply.github.com>
This PR updates the Edge Runtime to use a new version that loads dependencies differently. This addresses https://github.com/vercel/next.js/pull/38766 so `instanceof` works as expected.
It involved a few code changes, mostly regarding to types. The most important change is that the `Runner` function in the sandbox doesn't take a `ReadableStream` as `body` anymore since this implies creating the instance on "node land" and makes the runtime `fetch` function not to be able to compare with `ReadableStream` using `instanceof`. Instead we introduce a "clonable body" abstraction that allows to create the `ReadableStream` from `Readable` by using the edge runtime primitive which would hold the correct prototype.
Also, this PR changes the way we pre-compile the Edge Runtime to adapt it to the new version.
## What’s in there?
This PR brings more consistency in how errors and warnings are reported when running code in the Edge Runtime:
- Dynamic code evaluation (`eval()`, `new Function()`, `WebAssembly.instantiate()`, `WebAssembly.compile()`…)
- Usage of Node.js global APIs (`BroadcastChannel`, `Buffer`, `TextDecoderStream`, `setImmediate()`...)
- Usage of Node.js modules (`fs`, `path`, `child_process`…)
The new error messages should mention *Edge Runtime* instead of *Middleware*, so they are valid in both cases.
It also fixes a bug where the process polyfill would issue a warning for `process.cwd` (which is `undefined` but legit). Now, one has to invoke the function `process.cwd()` to trigger the error.
It finally fixes the react-dev-overlay, where links from middleware and Edge API route files could not be opened because of the `(middleware)/` prefix in their name.
About the later, please note that we can’t easily remove the prefix or change it for Edge API routes. It comes from the Webpack layer, which is the same for both. We may consider renaming it to *edge* instead in the future.
## How to test?
These changes are almost fully covered with tests:
```bash
pnpm testheadless --testPathPattern runtime-dynamic
pnpm testheadless --testPathPattern runtime-with-node
pnpm testheadless --testPathPattern runtime-module
pnpm testheadless --testPathPattern middleware-dev-errors
```
To try them out manually, you can write a middleware and Edge route files like these:
```jsx
// middleware.js
import { NextResponse } from 'next/server'
import { basename } from 'path'
export default async function middleware() {
eval('2+2')
setImmediate(() => {})
basename()
return NextResponse.next()
}
export const config = { matcher: '/' }
```
```jsx
// pages/api/route.js
import { basename } from 'path'
export default async function handle() {
eval('2+2')
setImmediate(() => {})
basename()
return Response.json({ ok: true })
}
export const config = { runtime: 'experimental-edge' }
```
The expected behaviours are:
- [x] dev, middleware/edge route is using a node.js module: error at runtime (logs + read-dev-overlay):
```bash
error - (middleware)/pages/api/route.js (1:0) @ Object.handle [as handler]
error - The edge runtime does not support Node.js 'path' module.
Learn More: https://nextjs.org/docs/messages/node-module-in-edge-runtime
> 1 | import { basename } from "path";
2 | export default async function handle() {
```
- [x] build, middleware/edge route is using a node.js module: warning but succeeds
```bash
warn - Compiled with warnings
./middleware.js
A Node.js module is loaded ('path' at line 4) which is not supported in the Edge Runtime.
Learn More: https://nextjs.org/docs/messages/node-module-in-edge-runtime
./pages/api/route.js
A Node.js module is loaded ('path' at line 1) which is not supported in the Edge Runtime.
Learn More: https://nextjs.org/docs/messages/node-module-in-edge-runtime
```
- [x] production, middleware/edge route is using a node.js module: error at runtime (logs + 500 error)
```bash
Error: The edge runtime does not support Node.js 'path' module.
Learn More: https://nextjs.org/docs/messages/node-module-in-edge-runtime
at <unknown> (file:///Users/damien/dev/next.js/packages/next/server/web/sandbox/context.ts:149)
```
- [x] dev, middleware/edge route is using a node.js global API: error at runtime (logs + read-dev-overlay):
```bash
error - (middleware)/pages/api/route.js (4:2) @ Object.handle [as handler]
error - A Node.js API is used (setImmediate) which is not supported in the Edge Runtime.
Learn more: https://nextjs.org/docs/api-reference/edge-runtime
2 |
3 | export default async function handle() {
> 4 | setImmediate(() => {})
| ^
```
- [x] build, middleware/edge route is using a node.js global API: warning but succeeds
```bash
warn - Compiled with warnings
./middleware.js
A Node.js API is used (setImmediate at line: 6) which is not supported in the Edge Runtime.
Learn more: https://nextjs.org/docs/api-reference/edge-runtime
./pages/api/route.js
A Node.js API is used (setImmediate at line: 3) which is not supported in the Edge Runtime.
Learn more: https://nextjs.org/docs/api-reference/edge-runtime
```
- [x] production, middleware/edge route is using a node.js module: error at runtime (logs + 500 error)
```bash
Error: A Node.js API is used (setImmediate) which is not supported in the Edge Runtime.
Learn more: https://nextjs.org/docs/api-reference/edge-runtime
at <unknown> (file:///Users/damien/dev/next.js/packages/next/server/web/sandbox/context.ts:330)
```
- [x] dev, middleware/edge route is loading dynamic code: warning at runtime (logs + read-dev-overlay) and request succeeds (we allow dynamic code in dev only):
```bash
warn - (middleware)/middleware.js (7:2) @ Object.middleware [as handler]
warn - Dynamic Code Evaluation (e. g. 'eval', 'new Function') not allowed in Edge Runtime
5 |
6 | export default async function middleware() {
> 7 | eval('2+2')
```
- [x] build, middleware/edge route is loading dynamic code: build fails with error:
```bash
Failed to compile.
./middleware.js
Dynamic Code Evaluation (e. g. 'eval', 'new Function', 'WebAssembly.compile') not allowed in Edge Runtime
Used by default
./pages/api/route.js
Dynamic Code Evaluation (e. g. 'eval', 'new Function', 'WebAssembly.compile') not allowed in Edge Runtime
Used by default
```
## Notes to reviewers
Edge-related errors are either issued from `next/server/web/sandbox/context.ts` file (runtime errors) or from `next/build/webpack/plugins/middleware-plugin.ts` (webpack compilation).
The previous implementation (I’m pleading guilty here) was way too verbose: some errors (Node.js global APIs like using `process.cwd()`) could be reported several times, and the previous mechanism to dedupe them (in middleware-plugin) wasn’t really effective.
Changes in tests are due to renaming existing tests such as `test/integration/middleware-with-node.js-apis` into `test/integration/edge-runtime-with-node.js-apis`. I extended them to cover Edge API route.
@hanneslund I’ve pushed the improvement you did in https://github.com/vercel/next.js/pull/38289/ one step further to avoid duplication.
* [edge] allow importing blob assets
* Fix test
* extract to a new file, to make it easier to read and review
* Use webpack asset discovery and transform with a loader
* fix tests
* don't prefix assets
* use emitFile
* rename assets to blobs to be more specific
* rename blobs to assets and use webpack's hashing algo
* Dedupe correctly
* Add a Node.js dep test
* Update packages/next/server/next-server.ts
Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
* [code review] test remote URL fetches
* [code review] use `import type` for type-only imports
* Update packages/next/server/next-server.ts
Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
* Apply suggestions from code review
Co-authored-by: JJ Kasper <jj@jjsweb.site>
Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
Co-authored-by: JJ Kasper <jj@jjsweb.site>
* Improve stack traces in dev mode
* Refactor `react-dev-overlay` to support the Edge Compiler
* Serialize errors including the compiler `source`
* Adopt the new `react-dev-overlay` displaying it for middleware errors
* Improve tests
* fix rsc cases
* update test
* use check for dev test
* handle different error from node version
Co-authored-by: feugy <damien@vercel.com>
Co-authored-by: JJ Kasper <jj@jjsweb.site>
This PR introduces [Edge Runtime](https://edge-runtime.vercel.app/) for emulating [Edge Functions](https://vercel.com/features/edge-functions) locally.
Every time you run a [middleware](https://nextjs.org/docs/advanced-features/middleware) locally via `next dev`, an isolated edge runtime context will be created.
These contexts have the same constraints as production servers, plus they don't pollute the global scope; Instead, all the code run in a vm on top of a Node.js process.
Additionally, `@edge-runtime/jest-environment` has been added to make easier testing Edge Functions in a programmatic way.
It dropped the following polyfills from Next.js codebase, since they are now part of Edge Runtime:
- abort-controller
- formdata
- uuid
- web-crypto
- web-streams
Co-authored-by: Gal Schlezinger <2054772+Schniz@users.noreply.github.com>
This PR brings some significant refactoring in preparation for upcoming middleware changes. Each commit can be reviewed independently, here is a summary of what each one does and the reasoning behind it:
- [Move pagesDir to next-dev-server](f2fe154c00) simply moves the `pagesDir` property to the dev server which is the only place where it is needed. Having it for every server is misleading.
- [Move (de)normalize page path utils to a file page-path-utils.ts](27cedf0871) Moves the functions to normalize and denormalize page paths to a single file that is intended to hold every utility function that transforms page paths. Since those are complementary it makes sense to have them together. I also added explanatory comments on why they are not idempotent and examples for input -> output that I find very useful.
- [Extract removePagePathTail](6b121332aa) This extracts a function to remove the tail on a page path (absolute or relative). I'm sure there will be other contexts where we can use it.
- [Extract getPagePaths and refactor findPageFile](cf2c7b842e) This extracts a function `getPagePaths` that is used to generate an array of paths to inspect when looking for a page file from `findPageFile`. Then it refactors such function to use it parallelizing lookups. This will allow us to print every path we look at when looking for a file which can be useful for debugging. It also adds a `flatten` helper.
- [Refactor onDemandEntryHandler](4be685c37e) I've found this one quite difficult to understand so it is refactored to use some of the previously mentioned functions and make it easier to read.
- [Extract absolutePagePath util](3bc0783474) Extracts yet another util from the `next-dev-server` that transforms an absolute path into a page name. Of course it adds comments, parameters and examples.
- [Refactor MiddlewarePlugin](c595a2cc62) This is the most significant change. The logic here was very hard to understand so it is totally redistributed with comments. This also removes a global variable `ssrEntries` that was deprecated in favour of module metadata added to Webpack from loaders keeping less dependencies. It also adds types and makes a clear distinction between phases where we statically analyze the code, find metadata and generate the manifest file cc @shuding @huozhi
EDIT:
- [Split page path utils](158fb002d0) After seeing one of the utils was being used by the client while it was defined originally in the server, with this PR we are splitting the util into multiple files and moving it to `shared/lib` in order to make explicit that those can be also imported from client.
This PR introduces a way to use WASM in middlewares.
Next.js will find all `.wasm` imports in middlewares and load them as `WebAssembly.Module` objects, which then can be later instantiated.
The metadata will be stored in `middleware-manifest.json`
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
Production middlewares will only expose env vars that are statically analyzable, as mentioned here: https://nextjs.org/docs/api-reference/next/server#how-do-i-access-environment-variables
This creates some incompatibility with `next dev` and `next start`, where all `process.env` data is shared and can lead to unexpected behavior in runtime.
This PR fixes it by limiting the data in `process.env` with the inferred env vars from the code usage. I believe the test speaks for itself 🕺
<!--
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
-->
**Note**: This PR is applying again changes landed #31935 that were reverted from an investigation.
This PR fixes#30398
By default Next will polyfill some fetch APIs (Request, Response, Header and fetch) only if fetch is not found in the global scope in certain entry points. If we have a custom server which is adding a global fetch (and only fetch) at the very top then the rest of APIs will not be polyfilled.
This PR adds a test on the custom server where we can add a custom polyfill for fetch with an env variable. This reproduces the issue since next-server.js will be required without having a polyfill for Response which makes it fail on requiring NextResponse. Then we remove the code that checks for subrequests to happen within the **sandbox** so that we don't need to polyfill `next-server` anymore.
The we also introduce an improvement on how we handle relative requests. Since #31858 introduced a `port` and `hostname` options for the server, we can always pass absolute URLs to the Middleware so we can always use the original `nextUrl` to pass it to fetch. This brings a lot of simplification for `NextURL` since we don't have to consider relative URLs no more.
## Bug
- [x] Related issues linked using `fixes #number`
- [x] Integration tests added
- [x] Errors have helpful link attached, see `contributing.md`
This PR fixes#30398
By default Next will polyfill some fetch APIs (Request, Response, Header and fetch) only if fetch is not found in the global scope in certain entry points. If we have a custom server which is adding a global fetch (and only fetch) at the very top then the rest of APIs will not be polyfilled.
This PR adds a test on the custom server where we can add a custom polyfill for fetch with an env variable. This reproduces the issue since next-server.js will be required without having a polyfill for Response which makes it fail on requiring NextResponse. Then we remove the code that checks for subrequests to happen within the **sandbox** so that we don't need to polyfill `next-server` anymore.
The we also introduce an improvement on how we handle relative requests. Since #31858 introduced a `port` and `hostname` options for the server, we can always pass absolute URLs to the Middleware so we can always use the original `nextUrl` to pass it to fetch. This brings a lot of simplification for `NextURL` since we don't have to consider relative URLs no more.
## Bug
- [x] Related issues linked using `fixes #number`
- [x] Integration tests added
- [x] Errors have helpful link attached, see `contributing.md`
To run middleware we are using a **sandbox** that emulates the web runtime and keeps a module cache. This cache is shared for all of the modules that we run using the sandbox while there are some module-level APIs that must be scoped depending on the module we are running.
One example of this is `fetch` where we want to always inject a special header that indicate the module that is performing the fetch and use it to avoid getting into infinite loops for middleware. For those cases the cached implementation will be the first one that instantiates the module and therefore we can actually get into infinite loops. This is the reason why #31800 is failing.
With this PR we refactor the sandbox so that the module cache is scoped per module name. This means that one execution of a middleware will preserve its cache only for that module so that each execution will still have its own `fetch` implementation, fixing this issue. Also, with this refactor the code is more clear and we also provide an option to avoid using the cache.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
This PR removes the class wrapper that we have for `TextEncoder` and `TextDecoder`. Since this is merged we will be using directly the Node version in the sandbox.
The `resource` argument[^1] in fetch may also be an instance of URL (or any other stringifiable value) but the sandbox variant of middlewares doesn't support that.
```js
export async function middleware(req, ev) {
await fetch(new URL('https://www.googleapis.com/oauth2/v3/certs'), {
redirect: 'manual',
method: 'GET',
})
return new Response(JSON.stringify({}), { status: 200 });
}
```
This is fixing the use of e.g. URL instance in `fetch`.
```
TypeError: initurl.startsWith is not a function
at getFetchURL (/my-next-app/node_modules/next/dist/server/web/sandbox/sandbox.js:246:17)
at fetch (/my-next-app/node_modules/next/dist/server/web/sandbox/sandbox.js:77:29)
at Object.middleware [as handler] (webpack-internal:///./pages/_middleware.js:86:15)
at async adapter (webpack-internal:///./node_modules/next/dist/server/web/adapter.js:30:22)
at async DevServer.runMiddleware (/my-next-app/node_modules/next/dist/server/next-server.js:430:26)
at async DevServer.runMiddleware (/my-next-app/node_modules/next/dist/server/dev/next-dev-server.js:394:28)
at async Object.fn (/my-next-app/node_modules/next/dist/server/next-server.js:807:34)
at async Router.execute (/my-next-app/node_modules/next/dist/server/router.js:211:32)
at async DevServer.run (/my-next-app/node_modules/next/dist/server/next-server.js:1115:29)
at async DevServer.run (/my-next-app/node_modules/next/dist/server/dev/next-dev-server.js:440:20)
```
[^1]: https://developer.mozilla.org/en-US/docs/Web/API/fetch#parameters
When libraries are required outside of the middleware function context and they do checks such as `a instanceof Uint8Array` since the constructors are different between the two contexts they'll always yield false.
This is a problem for libraries validating user input as well as the WebCryptoAPI polyfill used outside of Edge Functions.
- Fixes#30477
- Fixes#30911
This is only a problem for the sandbox runtime, not when ran inside an Edge Function.
Co-authored-by: Tobias Koppers <sokra@users.noreply.github.com>
With this PR we are updating the way we check the usage of `eval` and other dynamic code evaluation (like `new Function`) for middleware. Now instead of simply showing a warning it will behave differently depending on if we are building or in development.
- Development: we replace the dynamic code with a wrapper so that we print a warning only when the code is used. We don't fail in this scenario as it is possible that once the application is built the code that uses `eval` is left out.
- Build: we detect with tree shaking if the code that will be bundled into the middleware includes any dynamic code and in such scenario we make the build fail as don't want to allow it for the production environment.
Closes#30674
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
- Code splitting should be disabled for the server-web build. Done via `ServerlessPlugin`.
- ~Target can't be `web`, `webworker` is better.~ Using `web` and `es6` for now, still not ideal.
- https://github.com/acornjs/acorn/issues/970
## Bug
- [ ] Related issues linked using `fixes #number`
- [x] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
This PR adds support for [Middleware as per RFC ](https://github.com/vercel/next.js/discussions/29750).
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes
