This PR introduces [Edge Runtime](https://edge-runtime.vercel.app/) for emulating [Edge Functions](https://vercel.com/features/edge-functions) locally.
Every time you run a [middleware](https://nextjs.org/docs/advanced-features/middleware) locally via `next dev`, an isolated edge runtime context will be created.
These contexts have the same constraints as production servers, plus they don't pollute the global scope; Instead, all the code run in a vm on top of a Node.js process.
Additionally, `@edge-runtime/jest-environment` has been added to make easier testing Edge Functions in a programmatic way.
It dropped the following polyfills from Next.js codebase, since they are now part of Edge Runtime:
- abort-controller
- formdata
- uuid
- web-crypto
- web-streams
Co-authored-by: Gal Schlezinger <2054772+Schniz@users.noreply.github.com>
* Do not exclude internal _next request in middleware
* Allow for `NextURL` to parse prefetch requests
* Add test for middleware data prefetch
* Refactor `hasBasePath` and `replaceBasePath`
* Refactor `removeTrailingSlash`
* Refactor parsed next url to use `getNextPathnameInfo`
* Allow to configure `NextURL`
* Ensure middleware rewrites with always with a locale
Co-authored-by: JJ Kasper <jj@jjsweb.site>
_Hello Next.js team! First PR here, I hope I've followed the right practices._
### What's in there?
It has been decided to only support the following uses cases in Next.js' middleware:
- rewrite the URL (`x-middleware-rewrite` response header)
- redirect to another URL (`Location` response header)
- pass on to the next piece in the request pipeline (`x-middleware-next` response header)
1. during development, a warning on console tells developers when they are returning a response (either with `Response` or `NextResponse`).
2. at build time, this warning becomes an error.
3. at run time, returning a response body will trigger a 500 HTTP error with a JSON payload containing the detailed error.
All returned/thrown errors contain a link to the documentation.
This is a breaking feature compared to the _beta_ middleware implementation, and also removes `NextResponse.json()` which makes no sense any more.
### How to try it?
- runtime behavior: `HEADLESS=true yarn jest test/integration/middleware/core`
- build behavior : `yarn jest test/integration/middleware/build-errors`
- development behavior: `HEADLESS=true yarn jest test/development/middleware-warnings`
### Notes to reviewers
The limitation happens in next's web adapter. ~The initial implementation was to check `response.body` existence, but it turns out [`Response.redirect()`](https://github.com/vercel/next.js/blob/canary/packages/next/server/web/spec-compliant/response.ts#L42-L53) may set the response body (https://github.com/vercel/next.js/pull/31886). Hence why the proposed implementation specifically looks at response headers.~
`Response.redirect()` and `NextResponse.redirect()` do not need to include the final location in their body: it is handled by next server https://github.com/vercel/next.js/blob/canary/packages/next/server/next-server.ts#L1142
Because this is a breaking change, I had to adjust several tests cases, previously returning JSON/stream/text bodies. When relevant, these middlewares are returning data using response headers.
About DevEx: relying on AST analysis to detect forbidden use cases is not as good as running the code.
Such cases are easy to detect:
```js
new Response('a text value')
new Response(JSON.stringify({ /* whatever */ })
```
But these are false-positive cases:
```js
function returnNull() { return null }
new Response(returnNull())
function doesNothing() {}
new Response(doesNothing())
```
However, I see no good reasons to let users ship middleware such as the one above, hence why the build will fail, even if _technically speaking_, they are not setting the response body.
## Feature
- [x] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [x] Integration tests added
- [x] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [x] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [x] Make sure the linting passes by running `yarn lint`
Hello,
This is an iteration after first work at https://github.com/vercel/next.js/pull/36478.
What that PR missed is a way to just get a cookie value. Well, this PR adds two new things:
`cookies.get` returns the cookie value that could be `string | undefined`:
```js
const response = new NextResponse()
response.cookies.set('foo', 'bar', { path: '/test' })
const value = response.cookies.get('foo')
console.log(value) // => 'bar'
```
Additionally, if you want to know all the cookie details, you can use `cookies.getWithOptions`:
```js
const response = new NextResponse()
response.cookies.set('foo', 'bar', { path: '/test' })
const { value, options } response.cookies.getWithOptions('foo')
console.log(value) // => 'bar'
console.log(options) // => { Path: '/test' }
```
This PR introduces a more predictable API to manipulate cookies in an Edge Function context.
```js
const response = new NextResponse()
// set a cookie
response.cookies.set('foo, 'bar') // => set-cookie: 'foo=bar; Path=/'`
// set another cookie
response.cookies.set('fooz, 'barz') // => set-cookie: 'foo=bar; Path=/, fooz=barz; Path=/'`
// delete a cookie means mark it as expired
response.cookies.delete('foo') // => set-cookie: 'foo=; Path=/; Expires=Thu, 01 Jan 1970 00:00:00 GMT, fooz=barz; Path=/'`
// clear all cookies means mark all of them as expired
response.cookies.clear() // => set-cookie: 'fooz=; Path=/; Expires=Thu, 01 Jan 1970 00:00:00 GMT, foo=; Path=/; Expires=Thu, 01 Jan 1970 00:00:00 GMT'`
```
This new cookies API uses [Map](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map) interface, and it's available for `NextRequest` and `NextResponse`.
Additionally, you can pass a specific cookies option as a third argument in `set` method:
```js
response.cookies.set('foo', 'bar', {
path: '/',
maxAge: 60 * 60 * 24 * 7,
httpOnly: true,
sameSite: 'strict',
domain: 'example.com'
}
```
**Note**: `maxAge` it's in seconds rather than milliseconds.
Any cookie manipulation will be reflected over the `set-cookie` header, transparently.
closes#31719
This PR brings some significant refactoring in preparation for upcoming middleware changes. Each commit can be reviewed independently, here is a summary of what each one does and the reasoning behind it:
- [Move pagesDir to next-dev-server](f2fe154c00) simply moves the `pagesDir` property to the dev server which is the only place where it is needed. Having it for every server is misleading.
- [Move (de)normalize page path utils to a file page-path-utils.ts](27cedf0871) Moves the functions to normalize and denormalize page paths to a single file that is intended to hold every utility function that transforms page paths. Since those are complementary it makes sense to have them together. I also added explanatory comments on why they are not idempotent and examples for input -> output that I find very useful.
- [Extract removePagePathTail](6b121332aa) This extracts a function to remove the tail on a page path (absolute or relative). I'm sure there will be other contexts where we can use it.
- [Extract getPagePaths and refactor findPageFile](cf2c7b842e) This extracts a function `getPagePaths` that is used to generate an array of paths to inspect when looking for a page file from `findPageFile`. Then it refactors such function to use it parallelizing lookups. This will allow us to print every path we look at when looking for a file which can be useful for debugging. It also adds a `flatten` helper.
- [Refactor onDemandEntryHandler](4be685c37e) I've found this one quite difficult to understand so it is refactored to use some of the previously mentioned functions and make it easier to read.
- [Extract absolutePagePath util](3bc0783474) Extracts yet another util from the `next-dev-server` that transforms an absolute path into a page name. Of course it adds comments, parameters and examples.
- [Refactor MiddlewarePlugin](c595a2cc62) This is the most significant change. The logic here was very hard to understand so it is totally redistributed with comments. This also removes a global variable `ssrEntries` that was deprecated in favour of module metadata added to Webpack from loaders keeping less dependencies. It also adds types and makes a clear distinction between phases where we statically analyze the code, find metadata and generate the manifest file cc @shuding @huozhi
EDIT:
- [Split page path utils](158fb002d0) After seeing one of the utils was being used by the client while it was defined originally in the server, with this PR we are splitting the util into multiple files and moving it to `shared/lib` in order to make explicit that those can be also imported from client.
As the title. This is intended to be applied on both middleware and edge functions.
## Bug
- [ ] Related issues linked using `fixes #number`
- [x] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
Co-authored-by: Jiachi Liu <4800338+huozhi@users.noreply.github.com>
fixes https://github.com/vercel/next.js/issues/36136
## Bug
- [X] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
* attach pipeTo and pipeThrough polyfills to instance
* remove transformer polyfill
* remove polyfill
* fix missing polyfill in sandbox
* always polyfill the runtime
* always polyfill web streams in renderer
* fix missing AbortController and AbortSignal
* type fix
* fix type generation
* use global
Co-authored-by: Shu Ding <shu@shus-mac-studio.localdomain>
Something I noticed while using Middleware is that the older and no longer working methods are still accessible for the `event` with autocompletion because of their types, and the way of knowing they're deprecated is by running the app and seeing the error message, this should improve that when coding in tools like VS Code.
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [x] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
Co-authored-by: JJ Kasper <22380829+ijjk@users.noreply.github.com>
Moves two utility functions from `server/router.ts` into their own file. This avoids the middleware pulling in the full Next.js router into its bundle.
There are probably more opportunities like this, but this is a good start. Middleware should likely be bundled by a non-chunking optimizing compiler.
The `readableStreamTee` util is only used by server/render.tsx, but ended up in the middleware runtime bundle. So it's better to add all the utils in one place, and we can remove them once upstream implementation of ReadableStream is ready.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
* Fix the static streaming render result isn't the resolved streaming rendering content (resolve suspense)
* Update readable stream polyfill to fit the w3c standard
This PR introduces a way to use WASM in middlewares.
Next.js will find all `.wasm` imports in middlewares and load them as `WebAssembly.Module` objects, which then can be later instantiated.
The metadata will be stored in `middleware-manifest.json`
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
This PR brings back @Schniz awesome contribution to bring in bodies to middleware. It was reverted to leave it out of the stable release and to have some time to test it out in canary before officially releasing it. This PR is simply a `cherry-pick` of his original work.
Closes: #30953
Closes: https://github.com/vercel/next.js/pull/34490
Co-authored-by: Gal Schlezinger <2054772+Schniz@users.noreply.github.com>
Fixes the problem that global `process` variable has only the `env` field.
Also fixed the issue that the `env` field is empty when the `process` module is used as the value of the variable (which happens when the module is contained in a dependency of application).
## Bug
- [ ] Related issues linked using `fixes #number`
- [x] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
Since we're always using `ReadableStream`, we should just get rid of `ResultPiper`.
This also lets us replace things like `bufferedReadFromReadableStream` with a `TransformStream` that does the same thing, so that it's `TransformStream`s all the way down.
Finally, we can get rid of the one-off call to `renderToReadableStream` and just use `renderToStream` whenever we're rendering a concurrent tree.
Instead of branching rendering based on Node.js and browser/web runtimes, we should just use the web version for now, which can run as-is on versions >=16.5.0 of Node.js, polyfilling `ReadableStream` on older versions when necessary.
There are a few potential downsides to this, as React is less able to optimize flushing and execution. We can revisit that in the future though if desired.
* fix: update default redirect status to 307 to match docs
* fix: update default redirect status to 307 to match docs
Co-authored-by: JJ Kasper <jj@jjsweb.site>
We currently have inconsistencies when working with URLs in the Edge Functions runtime, this PR addresses them introducing a warning for inconsistent usage that will break in the future. Here is the reasoning.
### The Browser
When we are in a browser environment there is a fixed location stored at `globalThis.location`. Then, if one tries to build a request with a relative URL it will work using that location global hostname as _base_ to construct its URL. For example:
```typescript
// https://nextjs.org
new Request('/test').url; // https://nextjs.org/test
Response.redirect('/test').headers.get('Location'); // https://nextjs.org/test
```
However, if we attempt to run the same code from `about:blank` it would not work because the global to use as a base `String(globalThis.location)` is not a valid URL. Therefore a call to `Response.redirect('/test')` or `new Response('/test')` would fail.
### Edge Functions Runtime
In Next.js Edge Functions runtime the situation is slightly different from a browser. Say that we have a root middleware (`pages/_middleware`) that gets invoked for every page. In the middleware file we expose the handler function and also define a global variable that we mutate on every request:
```typescript
// pages/_middleware
let count = 0;
export function middleware(req: NextRequest) {
console.log(req.url);
count += 1;
}
```
Currently we cache the module scope in the runtime so subsequent invocations would hold the same globals and the module would not be evaluated again. This would make the counter to increment for each request that the middleware handles. It is for this reason that we **can't have a global location** that changes across different invocations. Each invocation of the same function uses the same global which also holds primitives like `URL` or `Request` so changing an hypothetical `globalThis.location` per request would affect concurrent requests being handled.
Then, it is not possible to use relative URLs in the same way the browser does because we don't have a global to rely on to use its host to compose a URL from a relative path.
### Why it works today
We are **not** validating what is provided to, for example, `NextResponse.rewrite()` nor `NextResponse.redirect()`. We simply create a `Response` instance that adds the corresponding header for the rewrite or the redirect. Then it is **the consumer** the one that composes the final destination based on the request. Theoretically you can pass any value and it would fail on redirect but won't validate the input.
Of course this is inconsistent because it doesn't make sense that `NextResponse.rewrite('/test')` works but `fetch(new NextRequest('/test'))` does not. Also we should validate what is provided. Finally, we want to be consistent with the way a browser behaves so `new Request('/test')` _should_ not work if there is no global location which we lack.
### What this PR does
We will have to deprecate the usage of relative URLs in the previously mentioned scenarios. In preparation for it, this PR adds a validation function in those places where it will break in the future, printing a warning with a link that points to a Next.js page with an explanation of the issue and ways to fix it.
Although middleware changes are not covered by semver, we will roll this for some time to make people aware that this change is coming. Then after a reasonable period of time we can remove the warning and make the code fail when using relative URLs in the previously exposed scenarios.
Part of #31506, this PR moves the code of middleware handling from the base server to the node server.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
Production middlewares will only expose env vars that are statically analyzable, as mentioned here: https://nextjs.org/docs/api-reference/next/server#how-do-i-access-environment-variables
This creates some incompatibility with `next dev` and `next start`, where all `process.env` data is shared and can lead to unexpected behavior in runtime.
This PR fixes it by limiting the data in `process.env` with the inferred env vars from the code usage. I believe the test speaks for itself 🕺
<!--
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
-->
It adds AbortController and AbortSignal Web runtimes APIs to be used by the user at Edge Functions.
For doing that it delegates into `abort-controller` dependency that has been frozen to prevent any modification.
Co-authored-by: Zhang Zhi <20026577+fytriht@users.noreply.github.com>
# Description
The redirect responses do not contain a message body. This is in conflict with the RFCs (below) and causes Traefik (a reverse proxy) to invalidate the responses. In this pull request, I add a response body to the redirect responses.
This PR is similar to https://github.com/vercel/next.js/pull/25257, it appears that there are some other locations where redirection is handled incorrectly in next.js.
# References
- https://datatracker.ietf.org/doc/html/rfc7230#section-3.3
> All 1xx (Informational), 204 (No Content), and 304 (Not Modified) responses must not include a message-body. All other responses do include a message-body, although the body may be of zero length.
- https://datatracker.ietf.org/doc/html/rfc7231#section-6.4.3
> The server's response payload usually contains a short hypertext note with a hyperlink to the different URI(s).
Co-authored-by: JJ Kasper <22380829+ijjk@users.noreply.github.com>
**Note**: This PR is applying again changes landed #31935 that were reverted from an investigation.
This PR fixes#30398
By default Next will polyfill some fetch APIs (Request, Response, Header and fetch) only if fetch is not found in the global scope in certain entry points. If we have a custom server which is adding a global fetch (and only fetch) at the very top then the rest of APIs will not be polyfilled.
This PR adds a test on the custom server where we can add a custom polyfill for fetch with an env variable. This reproduces the issue since next-server.js will be required without having a polyfill for Response which makes it fail on requiring NextResponse. Then we remove the code that checks for subrequests to happen within the **sandbox** so that we don't need to polyfill `next-server` anymore.
The we also introduce an improvement on how we handle relative requests. Since #31858 introduced a `port` and `hostname` options for the server, we can always pass absolute URLs to the Middleware so we can always use the original `nextUrl` to pass it to fetch. This brings a lot of simplification for `NextURL` since we don't have to consider relative URLs no more.
## Bug
- [x] Related issues linked using `fixes #number`
- [x] Integration tests added
- [x] Errors have helpful link attached, see `contributing.md`
This PR fixes#30398
By default Next will polyfill some fetch APIs (Request, Response, Header and fetch) only if fetch is not found in the global scope in certain entry points. If we have a custom server which is adding a global fetch (and only fetch) at the very top then the rest of APIs will not be polyfilled.
This PR adds a test on the custom server where we can add a custom polyfill for fetch with an env variable. This reproduces the issue since next-server.js will be required without having a polyfill for Response which makes it fail on requiring NextResponse. Then we remove the code that checks for subrequests to happen within the **sandbox** so that we don't need to polyfill `next-server` anymore.
The we also introduce an improvement on how we handle relative requests. Since #31858 introduced a `port` and `hostname` options for the server, we can always pass absolute URLs to the Middleware so we can always use the original `nextUrl` to pass it to fetch. This brings a lot of simplification for `NextURL` since we don't have to consider relative URLs no more.
## Bug
- [x] Related issues linked using `fixes #number`
- [x] Integration tests added
- [x] Errors have helpful link attached, see `contributing.md`
This ensures type checking passes correctly for middleware types when `skipLibCheck: false` is set in `tsconfig.json`. This also moves the `middleware-types` to be an isolated test to ensure it isn't relying on any monorepo dependencies.
## Bug
- [x] Related issues linked using `fixes #number`
- [x] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
Fixes: https://github.com/vercel/next.js/issues/31992#issuecomment-984174872
* fix(types): add missing ua type for NextRequest
add missing ua string type on NextRequest["ua"], returns full agent as string
* Update UserAgent type
Co-authored-by: JJ Kasper <jj@jjsweb.site>
This allows TypeScript users to have type safety for middleware functions.
- Closes#30490
Co-authored-by: Lee Robinson <9113740+leerob@users.noreply.github.com>
Co-authored-by: JJ Kasper <22380829+ijjk@users.noreply.github.com>
To run middleware we are using a **sandbox** that emulates the web runtime and keeps a module cache. This cache is shared for all of the modules that we run using the sandbox while there are some module-level APIs that must be scoped depending on the module we are running.
One example of this is `fetch` where we want to always inject a special header that indicate the module that is performing the fetch and use it to avoid getting into infinite loops for middleware. For those cases the cached implementation will be the first one that instantiates the module and therefore we can actually get into infinite loops. This is the reason why #31800 is failing.
With this PR we refactor the sandbox so that the module cache is scoped per module name. This means that one execution of a middleware will preserve its cache only for that module so that each execution will still have its own `fetch` implementation, fixing this issue. Also, with this refactor the code is more clear and we also provide an option to avoid using the cache.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
Previously `response.cookie(name, value, options)` would mutate the passed in `options` which lead to unexpected behaviour as described in #31666.
This PR clones the `options` argument before mutating it.
## Bug
- [x] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [x] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [x] Make sure the linting passes by running `yarn lint`
Since `localhost` is actually an alias for `127.0.0.1` that points to loopback, we should take that into consideration at `NextURL` when we handle local URLs.
The implementation is based on [is-localhost-url](https://github.com/Kikobeats/is-localhost-url); I added some tests over local URLs variations present at the library to ensure other variations are working fine.
Additionally, I refactor some things over the code to avoid doing the same twice and added some legibility that is always welcome when you are working with URLs stuff.
closes https://github.com/vercel/next.js/issues/31533
