This ensures we have the correct `cache-control` header when `revalidate` is used with `notFound: true` on `getStaticProps` or `getServerSideProps` pages.
## Bug
- [x] Related issues linked using `fixes #number`
- [x] Integration tests added
- [x] Errors have helpful link attached, see `contributing.md`
Fixes: https://app.clubhouse.io/vercel/story/23445
Updates the React 18 concurrent tests to support streaming, by using webdriver (instead of cheerio) to support dynamic updates, and being more explicit about SSR vs. hydrated content.
Also enables streaming support by setting `supportsDynamicHTML: true` in `pipe(...)` for non-bot user agents.
Our `Observable` use has gotten sufficiently complex that it makes sense to just use a 3rd party implementation and not worry about maintaining it ourselves. As a bonus, it doesn't rely on Node APIs.
Allows opting in to support for new concurrent features, like server-side Suspense.
**!!! DO NOT USE !!!**
This is highly experimental. We **will** be gating additional breaking changes behind this same flag.
**!!! DO NOT USE !!!**
Also resolves suspense for static pages (i.e. `getStaticProps` or `next build`/`next export`) since we can't currently support streaming for those cases anyway.
Adds `RenderResult`, replacing the `string` that `renderToHTML` used to return, with an `Observable`-like API that callers can use to subscribe and get a callback when chunks are available to flush, etc.
This is the last architectural change needed for streaming. There are, however, other things currently standing in the way of streaming. For example, it is common to mutate `res` in `getServerSideProps` to do routing work, or write headers, before fetching page data. This pattern effectively nullifies any advantages of streaming. I may do a follow-up PR that adds an experimental alternative for applications not using React 18, but the main purpose for this support is for Suspense and Server Components.
For that reason, there's no actual streaming here yet: instead we just flush a single chunk. A follow-up PR will add support for streaming suspense boundaries in React 18.
It's possible for `renderToHTML` to return `null` if `res.finished || res.headersSent` is `true` after `getInitialProps` or `getServerSideProps`. In such cases, we can't generate a valid `ResponseCacheEntry` or `ResponsePayload`, so we shouldn't try.
I took the opportunity to add an invariant for when we expected a cacheable response, but didn't get one. This could happen because Next.js or an application erroneously mutated the underlying `res` during the rendering of a page with `getStaticProps`. This shouldn't normally be possible because `res` isn't exposed in such cases, but it's theoretically possible with a custom server, so it seemed worth flagging.
This allows the user to adjust the LRU cache size according to the
application need:
- optimize the cache size based on average page HTML & JSON size
- disable the memory cache by setting the size to 0
The hard-coded default of 50MB has been moved from the code to the
default configuration object.
Fixes#21535
See also #27325
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [X] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [X] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [X] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes
Currently there is a lot of mutation in the Next.js Server and the checks for Locale are directly coded in the general request handler. Ideally, we should have a function where we just pass the request input (url + headers + config) and generate a bunch of metadata that analyzes it generating all metadata we might require for both the URL and i18n + basePath information.
This PR brings:
- A new parsing function `parseUrl` that joins parsing an absolute/relative URL into a data structure compatible with the Node parsing output but missing redundant properties.
- A wrapper `parseNextURL` that extends `parseUrl` analyzing `i18n` and `basePath` based on the provided configuration, url and headers. This function is pure and stateless so it can be used outside of the Next.js context.
- Types improvements and reuse.
- Refactors `next-server.ts` request handling using the above mentioned functions so that the code there just apply effects to the `req` object and the `parsedUrl.query` leaving the code much more straightforward.
- Refactors `getRouteRegex` decomposing in two different functions where `getParametrizedRoute` can be used to retrieve the serializable data that is used to generate the Regex.
How a page is rendered depends on whether or not we're streaming. For example, if we're just rendering to a `string` or we're generating a response for a crawler or other robot, we don't want React 18 to dynamically flush `<script>` tags to update Suspense boundaries as they resolve. Instead, we just want to wait for the full HTML to resolve and return a result similar to `renderToString`.
This is what `RequestContext` and the new/refactored `pipe` and `getStaticHTML` methods allow. They add a `requireStaticHTML` option that gets passed down. A follow-up PR will make sure this is `true` when serving a robot, and also ensure React is invoked appropriately.
* Add x-forward headers to external rewrites
This commit configures the proxy used for external rewrites to include
x-forward headers [1]. This is particularly useful for incremental
adoption, where some routes will be handled by Next.js and others by a
different website. For example, a Rails app will use the
X-Forwarded-Host header to determine which host to use for URL
generation and redirects [2].
[1]: 91fee3e943/lib/http-proxy.js (L31)
[2]: 41139f6ba2/actionpack/lib/action_dispatch/http/url.rb (L221-L227)
* Handle image-optimizer case
Co-authored-by: JJ Kasper <jj@jjsweb.site>
This updates redirects' regexes to not match `/_next` paths since this is currently unexpected and can easily cause a multi-match redirect to break loading client-side assets. This also fixes custom-routes not matching correctly when `trailingSlash: true/false` is used
## Bug
- [x] Related issues linked using `fixes #number`
- [x] Integration tests added
- [x] Errors have helpful link attached, see `contributing.md`
x-ref: https://github.com/vercel/next.js/discussions/24683
x-ref: [slack thread](https://vercel.slack.com/archives/CGU8HUTUH/p1626159845474000)
This adds some missing fields to the `NextConfig` type we expose under `next` and also adds a `NextConfigComplete` internal type that allows us to not treat all fields as optional like you would when using the type in `next.config.js`.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes
By itself, `withCoalescedInvoke` with a separate `this.incrementalCache.set(...)` isn't really suitable for streaming responses. Since streaming is asynchronous, updating the cache separately introduces a gap where another origin request for the same resource could be made.
This could potentially be addressed by moving the cache update, but then `IncrementalCache` itself would need to be made to support streaming, in addition to the many other responsibilities it has. In this case, it seemed best to just use composition to add another caching layer in front of it, which is a familiar and understandable concept. Eventually, we might want to move this cache to the HTTP layer, which will also be simpler with this change.
As an added bonus, `renderToResponseWithComponents` becomes significantly simpler, and we delete some duplication.
Make the typing for `IncrementalCache` more explicit. With streaming, we’ll want to stream page data as well as HTML. This is a bit complicated now because we’re overloading `pageData` for both redirects and pages.
This PR makes the different types explicit. With streaming, the data for redirects is synchronously available, while the data for pages will become a stream.
A follow up PR will add a “stream through” cache in front of `IncrementalCache`
Refactors the internals of `next-server` to use `ResponsePayload` instead of `string | null` and manual `sendPayload` calls. This is the first step toward streaming support.
I split `renderToResponseWithComponents` into a separate `renderToResponseWithComponentsInternal` function for ease of review: GitHub's diff rendering was highly misleading, making it seem as though more of the function had changed. The separate function just makes the actual change clearer: we split `renderToHTMLWithComponents` into two promises; one that represents the actual render result, and one that represents all of the work (including background work for e.g. revalidation) that needs to be done as part of generating the result.
These changes make it easier to bubble up a `ResponsePayload`, instead of sometimes calling `sendPayload` out-of-band, centralizing all payload handling in `sendResponse` and eventually a similar function for public APIs that return a string. This centralization will make it much easier to handle a response that needs to be streamed, which is coming soon in another PR.
This ensures when an error occurs in an API route while using minimal mode the error is bubbled so it can be handled at the top-level
## Bug
- [ ] Related issues linked using `fixes #number`
- [x] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
This solves the main use case from Issue #19914.
Previously, we would set the `Cache-Control` header to a constant and rely on the server cache. This would mean the browser would always request the image and the server could response with 304 Not Modified to omit the response body.
This PR changes the behavior such that the `max-age` will propagate from the upstream server to the Next.js Image Optimization Server and allow browser caching. ("upstream" meaning external server or just an internal route to an image)
This PR does not change the `max-age` for static imports which will remain `public, max-age=315360000, immutable`.
#### Pros:
- Fewer HTTP requests after initial browser visit
- User configurable `max-age` via the upstream image `Cache-Control` header
#### Cons:
- ~~Might be annoying for `next dev` when modifying a source image~~ (solved: use `max-age=0` for dev)
- Might cause browser to cache longer than expected (up to 2x longer than the server cache if requested in the last second before expiration)
## Bug
- [x] Related issues linked using `fixes #number`
