This refactors the `incremental-cache` and moves the file-system cache handling to it's own cache handler allowing it to be replaced by a custom cache handler (experimental).
Closes: https://github.com/vercel/next.js/pull/22619
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
- [ ] The examples guidelines are followed from [our contributing doc](https://github.com/vercel/next.js/blob/canary/contributing.md#adding-examples)
This PR brings some significant refactoring in preparation for upcoming middleware changes. Each commit can be reviewed independently, here is a summary of what each one does and the reasoning behind it:
- [Move pagesDir to next-dev-server](f2fe154c00) simply moves the `pagesDir` property to the dev server which is the only place where it is needed. Having it for every server is misleading.
- [Move (de)normalize page path utils to a file page-path-utils.ts](27cedf0871) Moves the functions to normalize and denormalize page paths to a single file that is intended to hold every utility function that transforms page paths. Since those are complementary it makes sense to have them together. I also added explanatory comments on why they are not idempotent and examples for input -> output that I find very useful.
- [Extract removePagePathTail](6b121332aa) This extracts a function to remove the tail on a page path (absolute or relative). I'm sure there will be other contexts where we can use it.
- [Extract getPagePaths and refactor findPageFile](cf2c7b842e) This extracts a function `getPagePaths` that is used to generate an array of paths to inspect when looking for a page file from `findPageFile`. Then it refactors such function to use it parallelizing lookups. This will allow us to print every path we look at when looking for a file which can be useful for debugging. It also adds a `flatten` helper.
- [Refactor onDemandEntryHandler](4be685c37e) I've found this one quite difficult to understand so it is refactored to use some of the previously mentioned functions and make it easier to read.
- [Extract absolutePagePath util](3bc0783474) Extracts yet another util from the `next-dev-server` that transforms an absolute path into a page name. Of course it adds comments, parameters and examples.
- [Refactor MiddlewarePlugin](c595a2cc62) This is the most significant change. The logic here was very hard to understand so it is totally redistributed with comments. This also removes a global variable `ssrEntries` that was deprecated in favour of module metadata added to Webpack from loaders keeping less dependencies. It also adds types and makes a clear distinction between phases where we statically analyze the code, find metadata and generate the manifest file cc @shuding @huozhi
EDIT:
- [Split page path utils](158fb002d0) After seeing one of the utils was being used by the client while it was defined originally in the server, with this PR we are splitting the util into multiple files and moving it to `shared/lib` in order to make explicit that those can be also imported from client.
x-ref: #36190
x-ref: #31506
* Move nodejs ptah module usage to next-server, keep base-server and web-server headless for `'path'`
* Use a native module `path` for nodejs runtime and `path` polyfill for edge runtime
(#31506 for context)
This PR implements the minimum viable web server on top of the Next.js base server, and integrates it into our middleware (edge) SSR runtime to handle all the requests.
This also addresses problems like missing dynamic routes support in our current handler.
Note that this is the initial implementation with the assumption that the web server is running under minimal mode. Also later we can refactor the `__server_context` environment to properly passing the context via the constructor or methods.
Part of #31506, this PR moves the code of middleware handling from the base server to the node server.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
This fixes revalidation not occurring correctly when `notFound: true` is returned during build, additional tests have been added to ensure this is working correctly for dynamic and non-dynamic pages returning `notFound: true` during build and then revalidating afterwards.
## Bug
- [x] Related issues linked using `fixes #number`
- [x] Integration tests added
- [x] Errors have helpful link attached, see `contributing.md`
Fixes: https://github.com/vercel/next.js/issues/21453
This allows the user to adjust the LRU cache size according to the
application need:
- optimize the cache size based on average page HTML & JSON size
- disable the memory cache by setting the size to 0
The hard-coded default of 50MB has been moved from the code to the
default configuration object.
Fixes#21535
See also #27325
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [X] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [X] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [X] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes
Fixes: #24806Fixes: #26689Fixes: #27325Closes: #24807
@tommarshall has done us a huge favor with his PR https://github.com/vercel/next.js/pull/24807 which outlines exactly the issue and a pragmatic solution.
I'm not trying to "steal their work", however, the PR seems to have been stuck for some months. I think there's huge value in this for myself and others as essentially **ISR is broken** for people running Next.js at scale 😱
This PR has cherry-picked @tommarshall's fine fix and added some integrations tests around page revalidation and the edge case when the cache size is exhausted.
✏️ Edits are enabled, so feel free great Vercel staff and other maintainers to improve my bad tests or surrounding code 🙇
## Bug
- [x] Related issues linked using `fixes #number`
- [x] Integration tests added
Make the typing for `IncrementalCache` more explicit. With streaming, we’ll want to stream page data as well as HTML. This is a bit complicated now because we’re overloading `pageData` for both redirects and pages.
This PR makes the different types explicit. With streaming, the data for redirects is synchronously available, while the data for pages will become a stream.
A follow up PR will add a “stream through” cache in front of `IncrementalCache`
