This PR introduces [Edge Runtime](https://edge-runtime.vercel.app/) for emulating [Edge Functions](https://vercel.com/features/edge-functions) locally.
Every time you run a [middleware](https://nextjs.org/docs/advanced-features/middleware) locally via `next dev`, an isolated edge runtime context will be created.
These contexts have the same constraints as production servers, plus they don't pollute the global scope; Instead, all the code run in a vm on top of a Node.js process.
Additionally, `@edge-runtime/jest-environment` has been added to make easier testing Edge Functions in a programmatic way.
It dropped the following polyfills from Next.js codebase, since they are now part of Edge Runtime:
- abort-controller
- formdata
- uuid
- web-crypto
- web-streams
Co-authored-by: Gal Schlezinger <2054772+Schniz@users.noreply.github.com>
This PR brings some significant refactoring in preparation for upcoming middleware changes. Each commit can be reviewed independently, here is a summary of what each one does and the reasoning behind it:
- [Move pagesDir to next-dev-server](f2fe154c00) simply moves the `pagesDir` property to the dev server which is the only place where it is needed. Having it for every server is misleading.
- [Move (de)normalize page path utils to a file page-path-utils.ts](27cedf0871) Moves the functions to normalize and denormalize page paths to a single file that is intended to hold every utility function that transforms page paths. Since those are complementary it makes sense to have them together. I also added explanatory comments on why they are not idempotent and examples for input -> output that I find very useful.
- [Extract removePagePathTail](6b121332aa) This extracts a function to remove the tail on a page path (absolute or relative). I'm sure there will be other contexts where we can use it.
- [Extract getPagePaths and refactor findPageFile](cf2c7b842e) This extracts a function `getPagePaths` that is used to generate an array of paths to inspect when looking for a page file from `findPageFile`. Then it refactors such function to use it parallelizing lookups. This will allow us to print every path we look at when looking for a file which can be useful for debugging. It also adds a `flatten` helper.
- [Refactor onDemandEntryHandler](4be685c37e) I've found this one quite difficult to understand so it is refactored to use some of the previously mentioned functions and make it easier to read.
- [Extract absolutePagePath util](3bc0783474) Extracts yet another util from the `next-dev-server` that transforms an absolute path into a page name. Of course it adds comments, parameters and examples.
- [Refactor MiddlewarePlugin](c595a2cc62) This is the most significant change. The logic here was very hard to understand so it is totally redistributed with comments. This also removes a global variable `ssrEntries` that was deprecated in favour of module metadata added to Webpack from loaders keeping less dependencies. It also adds types and makes a clear distinction between phases where we statically analyze the code, find metadata and generate the manifest file cc @shuding @huozhi
EDIT:
- [Split page path utils](158fb002d0) After seeing one of the utils was being used by the client while it was defined originally in the server, with this PR we are splitting the util into multiple files and moving it to `shared/lib` in order to make explicit that those can be also imported from client.
As the title. This is intended to be applied on both middleware and edge functions.
## Bug
- [ ] Related issues linked using `fixes #number`
- [x] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
Co-authored-by: Jiachi Liu <4800338+huozhi@users.noreply.github.com>
* attach pipeTo and pipeThrough polyfills to instance
* remove transformer polyfill
* remove polyfill
* fix missing polyfill in sandbox
* always polyfill the runtime
* always polyfill web streams in renderer
* fix missing AbortController and AbortSignal
* type fix
* fix type generation
* use global
Co-authored-by: Shu Ding <shu@shus-mac-studio.localdomain>
* Fix the static streaming render result isn't the resolved streaming rendering content (resolve suspense)
* Update readable stream polyfill to fit the w3c standard
This PR introduces a way to use WASM in middlewares.
Next.js will find all `.wasm` imports in middlewares and load them as `WebAssembly.Module` objects, which then can be later instantiated.
The metadata will be stored in `middleware-manifest.json`
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
Fixes the problem that global `process` variable has only the `env` field.
Also fixed the issue that the `env` field is empty when the `process` module is used as the value of the variable (which happens when the module is contained in a dependency of application).
## Bug
- [ ] Related issues linked using `fixes #number`
- [x] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
Since we're always using `ReadableStream`, we should just get rid of `ResultPiper`.
This also lets us replace things like `bufferedReadFromReadableStream` with a `TransformStream` that does the same thing, so that it's `TransformStream`s all the way down.
Finally, we can get rid of the one-off call to `renderToReadableStream` and just use `renderToStream` whenever we're rendering a concurrent tree.
Instead of branching rendering based on Node.js and browser/web runtimes, we should just use the web version for now, which can run as-is on versions >=16.5.0 of Node.js, polyfilling `ReadableStream` on older versions when necessary.
There are a few potential downsides to this, as React is less able to optimize flushing and execution. We can revisit that in the future though if desired.
Part of #31506, this PR moves the code of middleware handling from the base server to the node server.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
Production middlewares will only expose env vars that are statically analyzable, as mentioned here: https://nextjs.org/docs/api-reference/next/server#how-do-i-access-environment-variables
This creates some incompatibility with `next dev` and `next start`, where all `process.env` data is shared and can lead to unexpected behavior in runtime.
This PR fixes it by limiting the data in `process.env` with the inferred env vars from the code usage. I believe the test speaks for itself 🕺
<!--
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
-->
It adds AbortController and AbortSignal Web runtimes APIs to be used by the user at Edge Functions.
For doing that it delegates into `abort-controller` dependency that has been frozen to prevent any modification.
Co-authored-by: Zhang Zhi <20026577+fytriht@users.noreply.github.com>
**Note**: This PR is applying again changes landed #31935 that were reverted from an investigation.
This PR fixes#30398
By default Next will polyfill some fetch APIs (Request, Response, Header and fetch) only if fetch is not found in the global scope in certain entry points. If we have a custom server which is adding a global fetch (and only fetch) at the very top then the rest of APIs will not be polyfilled.
This PR adds a test on the custom server where we can add a custom polyfill for fetch with an env variable. This reproduces the issue since next-server.js will be required without having a polyfill for Response which makes it fail on requiring NextResponse. Then we remove the code that checks for subrequests to happen within the **sandbox** so that we don't need to polyfill `next-server` anymore.
The we also introduce an improvement on how we handle relative requests. Since #31858 introduced a `port` and `hostname` options for the server, we can always pass absolute URLs to the Middleware so we can always use the original `nextUrl` to pass it to fetch. This brings a lot of simplification for `NextURL` since we don't have to consider relative URLs no more.
## Bug
- [x] Related issues linked using `fixes #number`
- [x] Integration tests added
- [x] Errors have helpful link attached, see `contributing.md`
This PR fixes#30398
By default Next will polyfill some fetch APIs (Request, Response, Header and fetch) only if fetch is not found in the global scope in certain entry points. If we have a custom server which is adding a global fetch (and only fetch) at the very top then the rest of APIs will not be polyfilled.
This PR adds a test on the custom server where we can add a custom polyfill for fetch with an env variable. This reproduces the issue since next-server.js will be required without having a polyfill for Response which makes it fail on requiring NextResponse. Then we remove the code that checks for subrequests to happen within the **sandbox** so that we don't need to polyfill `next-server` anymore.
The we also introduce an improvement on how we handle relative requests. Since #31858 introduced a `port` and `hostname` options for the server, we can always pass absolute URLs to the Middleware so we can always use the original `nextUrl` to pass it to fetch. This brings a lot of simplification for `NextURL` since we don't have to consider relative URLs no more.
## Bug
- [x] Related issues linked using `fixes #number`
- [x] Integration tests added
- [x] Errors have helpful link attached, see `contributing.md`
To run middleware we are using a **sandbox** that emulates the web runtime and keeps a module cache. This cache is shared for all of the modules that we run using the sandbox while there are some module-level APIs that must be scoped depending on the module we are running.
One example of this is `fetch` where we want to always inject a special header that indicate the module that is performing the fetch and use it to avoid getting into infinite loops for middleware. For those cases the cached implementation will be the first one that instantiates the module and therefore we can actually get into infinite loops. This is the reason why #31800 is failing.
With this PR we refactor the sandbox so that the module cache is scoped per module name. This means that one execution of a middleware will preserve its cache only for that module so that each execution will still have its own `fetch` implementation, fixing this issue. Also, with this refactor the code is more clear and we also provide an option to avoid using the cache.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
This PR removes the class wrapper that we have for `TextEncoder` and `TextDecoder`. Since this is merged we will be using directly the Node version in the sandbox.
The `resource` argument[^1] in fetch may also be an instance of URL (or any other stringifiable value) but the sandbox variant of middlewares doesn't support that.
```js
export async function middleware(req, ev) {
await fetch(new URL('https://www.googleapis.com/oauth2/v3/certs'), {
redirect: 'manual',
method: 'GET',
})
return new Response(JSON.stringify({}), { status: 200 });
}
```
This is fixing the use of e.g. URL instance in `fetch`.
```
TypeError: initurl.startsWith is not a function
at getFetchURL (/my-next-app/node_modules/next/dist/server/web/sandbox/sandbox.js:246:17)
at fetch (/my-next-app/node_modules/next/dist/server/web/sandbox/sandbox.js:77:29)
at Object.middleware [as handler] (webpack-internal:///./pages/_middleware.js:86:15)
at async adapter (webpack-internal:///./node_modules/next/dist/server/web/adapter.js:30:22)
at async DevServer.runMiddleware (/my-next-app/node_modules/next/dist/server/next-server.js:430:26)
at async DevServer.runMiddleware (/my-next-app/node_modules/next/dist/server/dev/next-dev-server.js:394:28)
at async Object.fn (/my-next-app/node_modules/next/dist/server/next-server.js:807:34)
at async Router.execute (/my-next-app/node_modules/next/dist/server/router.js:211:32)
at async DevServer.run (/my-next-app/node_modules/next/dist/server/next-server.js:1115:29)
at async DevServer.run (/my-next-app/node_modules/next/dist/server/dev/next-dev-server.js:440:20)
```
[^1]: https://developer.mozilla.org/en-US/docs/Web/API/fetch#parameters
When libraries are required outside of the middleware function context and they do checks such as `a instanceof Uint8Array` since the constructors are different between the two contexts they'll always yield false.
This is a problem for libraries validating user input as well as the WebCryptoAPI polyfill used outside of Edge Functions.
- Fixes#30477
- Fixes#30911
This is only a problem for the sandbox runtime, not when ran inside an Edge Function.
Co-authored-by: Tobias Koppers <sokra@users.noreply.github.com>
With this PR we are updating the way we check the usage of `eval` and other dynamic code evaluation (like `new Function`) for middleware. Now instead of simply showing a warning it will behave differently depending on if we are building or in development.
- Development: we replace the dynamic code with a wrapper so that we print a warning only when the code is used. We don't fail in this scenario as it is possible that once the application is built the code that uses `eval` is left out.
- Build: we detect with tree shaking if the code that will be bundled into the middleware includes any dynamic code and in such scenario we make the build fail as don't want to allow it for the production environment.
Closes#30674
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
- Code splitting should be disabled for the server-web build. Done via `ServerlessPlugin`.
- ~Target can't be `web`, `webworker` is better.~ Using `web` and `es6` for now, still not ideal.
- https://github.com/acornjs/acorn/issues/970
## Bug
- [ ] Related issues linked using `fixes #number`
- [x] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
This PR adds support for [Middleware as per RFC ](https://github.com/vercel/next.js/discussions/29750).
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes
