This PR brings some significant refactoring in preparation for upcoming middleware changes. Each commit can be reviewed independently, here is a summary of what each one does and the reasoning behind it:
- [Move pagesDir to next-dev-server](f2fe154c00) simply moves the `pagesDir` property to the dev server which is the only place where it is needed. Having it for every server is misleading.
- [Move (de)normalize page path utils to a file page-path-utils.ts](27cedf0871) Moves the functions to normalize and denormalize page paths to a single file that is intended to hold every utility function that transforms page paths. Since those are complementary it makes sense to have them together. I also added explanatory comments on why they are not idempotent and examples for input -> output that I find very useful.
- [Extract removePagePathTail](6b121332aa) This extracts a function to remove the tail on a page path (absolute or relative). I'm sure there will be other contexts where we can use it.
- [Extract getPagePaths and refactor findPageFile](cf2c7b842e) This extracts a function `getPagePaths` that is used to generate an array of paths to inspect when looking for a page file from `findPageFile`. Then it refactors such function to use it parallelizing lookups. This will allow us to print every path we look at when looking for a file which can be useful for debugging. It also adds a `flatten` helper.
- [Refactor onDemandEntryHandler](4be685c37e) I've found this one quite difficult to understand so it is refactored to use some of the previously mentioned functions and make it easier to read.
- [Extract absolutePagePath util](3bc0783474) Extracts yet another util from the `next-dev-server` that transforms an absolute path into a page name. Of course it adds comments, parameters and examples.
- [Refactor MiddlewarePlugin](c595a2cc62) This is the most significant change. The logic here was very hard to understand so it is totally redistributed with comments. This also removes a global variable `ssrEntries` that was deprecated in favour of module metadata added to Webpack from loaders keeping less dependencies. It also adds types and makes a clear distinction between phases where we statically analyze the code, find metadata and generate the manifest file cc @shuding @huozhi
EDIT:
- [Split page path utils](158fb002d0) After seeing one of the utils was being used by the client while it was defined originally in the server, with this PR we are splitting the util into multiple files and moving it to `shared/lib` in order to make explicit that those can be also imported from client.
All links on git.io will stop redirecting after April 29, 2022.
- https://github.blog/changelog/2022-04-25-git-io-deprecation/
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
Part of #31506, this PR removes `loadEnvConfig` and `chalk` from the base server while keeping the same behavior for the node server.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
A middleware can work as a proxy intercepting requests and then performing a `fetch` to the destination adding headers to the request / response as a "man in the middle". When using `fetch` from a middleware we are not in the context of a browser so we can't really use relative URLs, they must be always absolute.
Now consider the previous case when middleware is running in *server mode*. Typically in order to know the host where we are fetching we can use the `request.nextUrl` which is given to the middleware but in this case the invoker (which is next-server) has no context of the hostname, nor the port. To solve this use case we must make the invoker of the middleware aware of the origin hostname and port.
This PR:
- Introduces `hostname` and `port` as options for `NextServer`.
- Refactors types in `NextServer` and `NextDevServer` moving type only imports to the top of the file.
- Refactors `startServer` to do a best guess on the `hostname` and `port`, passing them down.
- Exposes `.port` and `.hostname` to be retrieved from the `app`.
In an upcoming PR we will pass the host guess to the middleware to solve the relative URL issue.
Generally, AVIF quality can be lower compared to WebP so we can adjust this for the user so that it looks roughly the same depending on if the browser supports AVIF or WebP.
- Fixes#31254
- Related to https://github.com/lovell/sharp/issues/2850
* Automatically using different port to start dev server for non-explicit port
* return correct port
* more reasonable changes
* fix isExplicitPort
* 1. rename isExplicitPort to allowRetry 2.restrict the number of retries
Co-authored-by: Steven <steven@ceriously.com>
This replaces the server-sent events HMR connection with a WebSocket connection to prevent hitting browser connection limits, allow sending events back from the browser, and overall better performance.
This approach sets up the the `upgrade` event listener on the server immediately when created via `next dev` and on the first request using `req.socket.server` when created via a custom server. In a follow-up PR we can push the files changed via the WebSocket as well.
x-ref: https://github.com/vercel/next.js/issues/10061
x-ref: https://github.com/vercel/next.js/issues/8064
x-ref: https://github.com/vercel/next.js/issues/4495
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes
Bump `squoosh` to the latest version, currently commit [cad09160](cad09160b6).
Ideally, we would use the version published to npm but it hasn't been published in [two months](https://www.npmjs.com/package/@squoosh/lib?activeTab=versions) and we have a patch (#23565) that isn't available upstream.
This also is a precursor to getting support for AVIF.
- Fixes#27092
- Fixes#26527
- Reapplies the patch from #23565
This PR is a workaround for #24421 by adding an artificial delay when Apple M1 + buggy Node.js is detected.
Node.js 14 is unaffected because it M1 still reports `arch=x64`. Starting in Node.js 16, M1 reports `arch=arm64`.
V8 Bug: https://crbug.com/1224882
Node.js Issue: https://github.com/nodejs/node/issues/39327
This upgrades to ncc@0.25.0 and fixes the previous bugs including:
* ncc not referenced correctly in build
* Babel type errors
* node-fetch, etag, chalk and raw-body dependencies not building with ncc - these have been "un-ncc'd" for now. As they are relatively small dependencies, this doesn't seem too much of an issue and we can follow up in the tracking ncc issue at https://github.com/vercel/ncc/issues/612.
* `yarn dev` issues
Took a lot of bisecting, but the overall diff isn't too bad here in the end.
Disambiguate between pages/index.js and pages/index/index.js so that they resolve differently.
It all started with a bug in pagesmanifest that propagated throughout the codebase. After fixing pagesmanifest I was able to remove a few hacks here and there and more logic is shared now. especially the logic that resolves an entrypoint back into a route path. To sum up what happened:
- `getRouteFromEntrypoint` is the inverse operation of `getPageFile` that's under `pages/_document.tsx`
- `denormalizePagePath` is the inverse operation of `normalizePagePath`.
Everything is refactored in terms of these operations, that makes their behavior uniform and easier to update/patch in a central place. Before there were subtle differences between those that made `index/index.js` hard to handle.
Some potential follow up on this PR:
- [`hot-reloader`](https://github.com/vercel/next.js/pull/13699/files#diff-6161346d2c5f4b7abc87059d8768c44bR207) still has one place that does very similar behavior to `getRouteFromEntrypoint`. It can probably be rewritten in terms of `getRouteFromEntrypoint`.
- There are a few places where `denormalizePagePath(normalizePagePath(...))` is happening. This is a sign that `normalizePagePath` is doing some validation that is independent of its rewriting logic. That should probably be factored out in its own function. after that I should probably investigate whether `normalizePagePath` is even still needed at all.
- a lot of code is doing `.replace(/\\/g, '')`. If wanted, that could be replaced with `normalizePathSep`.
- It looks to me like some logic that's spread across the project can be centralized in 4 functions
- `getRouteFromEntrypoint` (part of this PR)
- its inverse `getEntrypointFromRoute` (already exists in `_document.tsx` as `getPageFile`)
- `getRouteFromPageFile`
- its inverse `getPageFileFromRoute` (already exists as `findPageFile ` in `server/lib/find-page-file.ts`)
It could be beneficial to structure the code to keep these fuctionalities close together and name them similarly.
- revise `index.amp` handling in pagesmanifest. I left it alone in this PR to keep it scoped, but it may be broken wrt nested index files as well. It might even make sense to reshape the pagesmanifest altogether to handle html/json/amp/... better
* fix(debugging): do not pass NODE_OPTIONS='--inspect' to subprocesses
fixes#11030
* fix(debugger): use a regex to remove bad NODE_OPTIONS flags
Co-authored-by: Alec Larson <alec.stanford.larson@gmail.com>
* Convert Dev Server to TypeScript
This converts the Next.js Development Server to TypeScript in prep for upcoming changes.
* Convert remaining necessary
* Fix some type errors
* Adjust types
* Fail build on duplicate pages
This will fail the `next build` command when a duplicate page is found.
In development, we'll emit a warning instead of crashing the dev server.
* Add test for warning in development
* Only issue a warning
* Fix production test
* Fix development test
* Remove useless arg
* Warn in development, too
* Run prettier over packages/**/*.js
* Run prettier over packages/**/*.ts
* Run prettier over examples
* Remove tslint
* Run prettier over examples
* Run prettier over all markdown files
* Run prettier over json files
* Add WithAmp to enable AMP support for
pages instead of .amp.js
* Update handling for exporting AMP
* Fix ampPath in export for / path and
revert isAmp logic to handle right
* Update amphtml test suite
* Add handling for noDirtyAmp during
export and update amp-export test suite
* Update serverless and export-default-map
test suites
* Update require-page tests
* Add support for .amp.js pages and
resolving /page?amp=1 to page.amp.js
* Update amp tests
* Update example and clean up amp page resolving
* Add nested amp test
* page => normalizedPage
* Add type to page options
* Add handling of amp with all pageExtensions
and normalize page
* Make sure findPageFile only falls back to
amp if enabled
We don't use a lot of the features of `glob`, so let's remove it in favor of a leaner approach using regex.
It's failing on windows and I have no idea why and don't own a windows machine 🤦🏼♂️
(Ignore some of the commits in here, I forgot to create the new branch before I started working)
* fix(launch-editor): resolve filename including current working directory
* refactor: handle when fileName is already absolute
* feat: use Next.js dir instead of process.cwd()
As I detailed in [this thread on Spectrum](https://spectrum.chat/?t=3df7b1fb-7331-4ca4-af35-d9a8b1cacb2c), the dev experience would be a lot nicer if the server started listening as soon as possible, before the slow initialization steps. That way, instead of manually polling the dev URL until the server's up (this can take a long time!), I can open it right away and the responses will be delivered when the dev server is done initializing.
This makes a few changes to the dev server:
* Move `HotReloader` creation to `prepare`. Ideally, more things (from the non-dev `Server`) would be moved to a later point as well, because creating `next({ ... })` is quite slow.
* In `run`, wait for a promise to resolve before doing anything. This promise automatically gets resolved whenever `prepare` finishes successfully.
And the `next dev` and `next start` scripts:
* Since we want to log that the server is ready/listening before the intensive build process kicks off, we return the app instance from `startServer` and the scripts call `app.prepare()`.
This should all be backwards compatible, including with all existing custom server recommendations that essentially say `app.prepare().then(listen)`. But now, we could make an even better recommendation: start listening right away, then call `app.prepare()` in the `listen` callback. Users would be free to make that change and get better DX.
Try it and I doubt you'll want to go back to the old way. :)
