Commit graph

162 commits

Author SHA1 Message Date
Tobias Koppers
381badcde3 Improvements to benchmarks (vercel/turbo#99)
* remove 100 modules form the default modules count
* rename bench_restart -> bench_hydration_cached
* add bench_startup_cached
* run npm install only once per test case and copy the result instead
* use multi-threaded copy
* Run `cached` tests only when `TURBOPACK_BENCH_CACHED` is set
* add `TURBOPACK_BENCH_BENCH` too measure the time it takes to run the
full benchmark (including startup and teardown) instead.
* add `TURBOPACK_BENCH_HEAD` to run non-headless
* add `TURBOPACK_BENCH_DEVTOOLS` to auto open devtools
* retry initial warmup change when it fails to detect that

This should make the benchmarks about 3 times faster... at least on
windows^^
2022-10-19 13:39:05 +02:00
Tobias Koppers
06ee2efe8e remove get_by_id (for performance) (vercel/turbo#88)
I also needed to remove the old html runtime, use new runtime for CSP to
update the CSS HMR code
2022-10-19 07:33:58 +02:00
Leah
0f40ec8327 ts HMR runtime (vercel/turbo#38) 2022-10-18 20:37:33 +02:00
Will Binns-Smith
50ab6dab18 [Rebased] Remove automatic core-js polyfills (vercel/turbo#91)
Rebase of vercel/turbo#68, which was merged into a now-closed PR.

Closes vercel/turbo#65.

When configured to use preset-env, this stops turbopack from inserting
dependencies on core-js. It's:

* Buggy/error-prone, at least through swc preset-env: 
  * https://github.com/vercel/the-three-body/issues/65
* setImmediate "polyfills" always inserted for this nonstandard feature
when
    React (scheduler, iirc) does feature detection of it
* Big. Updating the snapshot alone for a test that only used [].includes
  resulted in a diff with nearly 4700 line deletions
* Not done by Next.js for modern targets in the past couple of years:
https://twitter.com/timneutkens/status/1234548900272517120

Test Plan: Added [].includes to preset_env snapshot test and verified no
references to core-js are inserted.
2022-10-18 11:04:41 -07:00
Alex Kirszenberg
7bf4a27e6a Ensure Next.js uses React 18 APIs, enable styled_jsx transform in SSR (vercel/turbo#90)
This will remove the warning about using `hydrate` instead of
`hydrateRoot`, and fix a mismatch between SSR and CSR when using
styled-jsx (`<style jsx>`).
2022-10-18 19:30:46 +02:00
Alex Kirszenberg
9062ad91f4 Add polyfills to Node.js externals on the client-side (vercel/turbo#89)
We can't use the normal `ImportMap` because we stop resolving if looking
up a request within yields a result. However, for module polyfills, we
only need to look up the request when normal resolving fails. Hence the
dichotomy between the `import_map` and the `fallback_import_map`.

In a more modular architecture, we would model this as a stack of three
resolvers:

1. ImportMapResolver
2. DefaultResolver
3. FallbackImportMapResolver

I'm guessing this is what we'll have in the future, but this requires a
bigger refactoring than what I'm comfortable with implementing for now.
2022-10-18 19:30:07 +02:00
Alex Kirszenberg
59b7290933 Remove unused pages/_app and pages/_document (vercel/turbo#78)
These are no longer used, as we are now using Next.js' own components
instead.
2022-10-18 14:53:01 +02:00
Tobias Koppers
105ff16578 remove ContentSource::vary to avoid many tasks (vercel/turbo#71) 2022-10-18 13:00:10 +02:00
Justin Ridgewell
feb8c9ac39 Implement support for static assets directory (vercel/turbo#73)
Fixes https://github.com/vercel/web-tooling-internal/issues/7

Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
2022-10-18 10:05:37 +02:00
Tobias Koppers
5c1d113f1e node.js needs a fetch polyfill (vercel/turbo#58) 2022-10-18 09:56:24 +02:00
Tobias Koppers
aed7fa4d0b add basic support for browser alias field (vercel/turbo#56) 2022-10-18 03:17:06 +02:00
Leah
fcc806e1e5 embedded next.js package (vercel/turbo#37) 2022-10-17 23:33:25 +02:00
Alex Kirszenberg
44c2e1f570 Optimize Next.js SSR (vercel/turbo#48)
This PR also includes the changes in vercel/turbo#12 

This optimizes Next.js SSR by marking `react[/*]` and `next[/*]` imports
as externals in Node.js, which means that they will be `require`d
directly instead of bundled.

This optimization reduces the number of modules to analyze on the server
for a single component app from ~1000 (Next.js SSR imports a lot of
modules, including amp-optimizer which includes more) to <10.

Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
2022-10-17 19:54:19 +02:00
Tobias Koppers
2f3edac20c add introspection for assets and content sources (vercel/turbo#47)
add very simple explorer to introspect the graph via `/__turbopack__/`

Allows to "walk" through the content sources and asset graph. Some
assets allows to access inner assets, e. g. from the ecmascript chunk
you can access the entry module asset, so you can go a level deeper.

It works by a `Introspectable` trait which can be implemented by
anything we want to allow the user to inspect. In future the in browser
UI should offer a nice UI to explore your application.

In future we probably also want to add some `metrics` to the
`Introspectable` trait, e. g. to report size of an asset.

This UI is very basic currently, but that works for debugging:


![image](https://user-images.githubusercontent.com/1365881/196007862-2fb2a0e9-19ba-45c5-8a82-926418e0d479.png)
2022-10-17 17:19:29 +02:00
Tobias Koppers
3c821767ce use browser field for the browser (vercel/turbo#26)
add module field
2022-10-17 08:36:30 +02:00
Tobias Koppers
cabb72fa53 avoid emitting source maps for node.js (vercel/turbo#30) 2022-10-16 10:39:18 +02:00
Leah
f39152cef6 add From<T> impl for File (vercel/turbo#35)
transferred from https://github.com/vercel/turbo-tooling/pull/519
2022-10-15 18:42:14 +02:00
Leah
003a1ed013 add FileSystem::root() function to replace FileSystemPathVc::new(fs, path) (vercel/turbo#34)
transferred from https://github.com/vercel/turbo-tooling/pull/518
2022-10-15 18:40:02 +02:00
Tobias Koppers
280e909131 Deduplicate issues for logging (vercel/turbo#27)
see https://github.com/vercel/turbo-tooling/pull/523

Co-authored-by: Justin Ridgewell <justin@ridgewell.name>
2022-10-14 17:52:59 +02:00
Tobias Koppers
8cab8aa1d6 windows needs magic SystemRoot env var to allow to make http requests from node.js (vercel/turbo#22) 2022-10-14 00:45:39 +02:00
Tobias Koppers
26b3d1578b Merge pull request vercel/turbo#10 from vercel/bugfix/deterministic
make less things to rely on HashSet and HashMap order
2022-10-13 18:19:46 +02:00
Tobias Koppers
79ea25f491 make less things to rely on HashSet and HashMap order 2022-10-13 18:16:12 +02:00
Tobias Koppers
4760f0dbfd improve startup error messages (vercel/turbo#524) 2022-10-13 23:38:19 +08:00
Alex Kirszenberg
0527d39665 Turn the default Document component into a class component (vercel/turbo#512)
Some code in userland can expect Document to be a class, and as such try to extend it. Extending a function will result in an invalid React component, so it must be a class component.

e.g. https://nextjs.org/docs/advanced-features/custom-document#typescript
2022-10-13 09:29:20 +08:00
Alex Kirszenberg
1d46acd4d4 Introduce PathRegex for patch matching with named parameters (vercel/turbo#511)
This is another attempt at fixing https://github.com/vercel/turbo-tooling/pull/507. It also handles the root index page properly ("pages/index.js").
2022-10-13 09:29:19 +08:00
Tobias Koppers
a28aead85b use react-refresh-utils bundled with next when available (vercel/turbo#509) 2022-10-13 09:29:19 +08:00
Tobias Koppers
d5ed7e3940 routing bugfixes (vercel/turbo#495) 2022-10-11 10:23:10 +00:00
Tobias Koppers
8589c15b95 avoid creating too many content sources (vercel/turbo#496) 2022-10-11 10:23:00 +00:00
Tobias Koppers
fb1629c542 continue preparations for .module.css (vercel/turbo#494)
split global css and module css into separate assets
2022-10-11 09:47:43 +00:00
Tobias Koppers
5e1922d397 move node.js rendering to a content source (vercel/turbo#486)
add getServerSideProps and dynamic segments

fixes vercel/web-tooling-internal#37
fixes vercel/turbo#444
2022-10-10 22:21:30 +00:00
Tobias Koppers
3a05f60d3c fix many root sources crated (vercel/turbo#485)
It's important to not create a `cell` outside of the root `source` method as this will create a new cell for each call, which means each `source` will create a separate task, and this bubbles for each cell created in `source`...
2022-10-10 13:44:42 +00:00
Alex Kirszenberg
7ebb64846c Make embed_file relative to package instead of invoker file (vercel/turbo#484)
This makes the argument to `embed_file!` relative to the package it is invoked from, instead of relative to the file it is invoked from. The latter is closer to how `include_*!` works, but it has a couple of issues:
* Since `file!` returns a path relative to the workspace, it requires knowing the full path to the workspace. However, this is undefined when the package is compiled from outside of the workspace (e.g. as a dependency of another package).
* Procedural macros don't have access to the path of the file where they were originally invoked [(although we could get it with #[feature(proc_macro_span)])](https://doc.rust-lang.org/proc_macro/struct.Span.html), which means that an `embed_dir` macro would not be able to be relative to the invoker file anyway. So this makes things more consistent for such a (to be implemented) macro.
2022-10-10 13:35:42 +00:00
Will Binns-Smith
9bc858fddd preset_env followup [1/2] (vercel/turbo#478)
Addresses some feedback from vercel/turbo#380. Together with a following PR to use swc_core (currently blocked on https://github.com/swc-project/swc/pull/6083 and a subsequent publish), will address vercel/turbo#474.

* Remove unnecessary Result type
* Remove unnecessary @swc/helpers
2022-10-10 14:46:36 +02:00
Alex Kirszenberg
ed769521f9 Support for App.getInitialProps/Page.getInitialProps (vercel/turbo#464)
This is a very simplified version of what Next.js does. It adds support for `App` and page components defining `getInitialProps`.

fixes vercel/web-tooling-internal#38
2022-10-10 09:03:25 +00:00
Justin Ridgewell
4851e09ca8 Revamp errors and logging (vercel/turbo#467)
This updates all of our Issues to have a consistent display, and changes the way we display source code and debug-level details. In particular, this makes the following changes:

1. Cleans up the title/description of existing issues
   - We were filling the `title` field with too much information, because the `desccription` field wasn't always shown.
1. Always displays the `description`
   - This used to be hidden behind the `--detail` flag
1. Always displays the code frame of the issue, if it exists
   - This used to be hidden behind the `--detail` flag
1. Fixes the `context` (file path) to display the relative path to the file
   - It was hardcoded to the FS string used by turbotrace
1. Creates a new `detail` function which can provide debug-level information about the internal Rust state
   - This is not shown by default, user must pass `--detail` flag

The format for a rendered issue roughly matches:

```
${SEVERITY} [${CATEGORY}]
  ${CONTEXT}
    ${TITLE}
    ${CODE FRAME}
    ${DESCRIPTION}
    ${DETAIL}
```

Where
- `SEVERITY` is error/warning/etc
- `CATEGORY` is a self-defined string for the phase of the compiler (eg, "parsing", "running")
- `CONTEXT` is a file path to the offending file
- `TITLE` is a short (single-line) title
- `CODE_FRAME` is an optional position that generated the issue, which will display a few context lines and highlight the offending span of code
- `DESCRIPTION` is a longer (multi-line) message that describes the issue, and what to do about it
- `DETAIL` is a (by default not shown) fully detailed, debug-level, message providing insights into the internal state of Rust
2022-10-10 06:24:54 +00:00
Tobias Koppers
9427514d69 prepare css for different types and transforms (vercel/turbo#477)
following the design of ecmascript parse and modules

fix a few nitpicks
2022-10-09 11:47:33 +00:00
Alex Kirszenberg
c3b6b0f790 Add support for pages/_document (vercel/turbo#456)
This adds preliminary support for pages/_document. The API is more complex than pages/_app, so this required a few more things in order to work.

Right now, our `next/document` shim does much, much less than the original. We will eventually have to figure out whether we should distribute our own module, or find a way to reuse Next.js' internals. There's *a lot* of stuff in there so the second option is probably best.
2022-10-07 12:49:09 +00:00
Alex Kirszenberg
839b55ff06 Add support for pages/_app (vercel/turbo#442)
This adds preliminary support for pages/_app. There's still some work to do here to support things like `App. getInitialProps`, etc., but that will come in later PRs. I've created issues for these for now.

One issue remaining is that renaming `pages/_app.js` does not switch to the default _app asset. The server needs to be restarted for this to happen. Similarly, adding an _app.js file does not switch from the default asset to it. I'm guessing there's something wrong with the way I set up the import mappings, and they aren't being invalidated properly?

fixes vercel/turbo#424
2022-10-07 11:36:44 +00:00
Tobias Koppers
39663ebc78 pass request info to content source (vercel/turbo#458)
allow the ContentSource to declare which info is required
2022-10-07 13:35:35 +02:00
Tobias Koppers
e8359f7964 refactor AssetContext (vercel/turbo#465)
remove context_path from AssetContext
remove most with_* methods from AssetContext
add ResolveOrigin trait
use ResolveOrigin to pass along context path and context for resolving
ecmascript and css module assets implement ResolveOrigin

Noteable change: Many places use `origin_path` instead of `context_path` which means it points to the issuer module path instead of the issuer module directory.
2022-10-07 13:34:31 +02:00
Will Binns-Smith
e4f92b7206 Automatic downleveling to browser targets using swc_preset_env (vercel/turbo#380)
Closes vercel/turbo#440

This uses swc's preset_env to automatically downlevel code according to the environment's browser targets, and sets next-dev to use a limited, modern target.

To do:
* [x] Add snapshot test
* [x] turbotrace test failures — looks like this has something to do with the Buffer module? Hitting swc's "multiple constructors" not implemented: f655488cfa/crates/swc_ecma_transforms_compat/src/es2015/classes/mod.rs (L545)
* [x] ~Benchmark downleveling node_modules (probably on front) and make a decision re: downleveling everything vs. just workspaces~ Filed as vercel/turbo#457
2022-10-07 06:54:21 +00:00
Tobias Koppers
d49a9c3b9d move ProcessEnv into turbo-tasks-env (vercel/turbo#462)
make ProcessEnv a trait to allow other implemenetations of it
2022-10-06 20:36:36 +02:00
LongYinan
8c80ba4f78 Migrate to pnpm (vercel/turbo#336)
### turbo tracing in pnpm projects

How pnpm link packages:

![image](https://user-images.githubusercontent.com/3468483/191022172-962af095-9b44-4b69-8e2f-1e8d440d1c31.png)

After turbo tracing:

```
dist
├── crates
│   └── turbopack
│       └── tests
│           └── node-file-trace
│               ├── integration
│               └── node_modules
│                   └── better-sqlite3 -> ../../../../../node_modules/.pnpm/better-sqlite3@7.6.2/node_modules/better-sqlite3
└── node_modules
    └── .pnpm
        ├── better-sqlite3@7.6.2
        │   └── node_modules
        │       ├── better-sqlite3
        │       │   ├── build
        │       │   │   └── Release
        │       │   └── lib
        │       │       └── methods
        │       └── bindings -> ../../bindings@1.5.0/node_modules/bindings
        ├── bindings@1.5.0
        │   └── node_modules
        │       ├── bindings
        │       └── file-uri-to-path -> ../../file-uri-to-path@1.0.0/node_modules/file-uri-to-path
        └── file-uri-to-path@1.0.0
            └── node_modules
                └── file-uri-to-path
```

No more **global .pnpm store**. The directory link will follow the original path created by pnpm.
2022-10-05 16:42:55 +00:00
Tobias Koppers
a8b0fd4f4b add layer to ChunkingContext and include that in module id and chunk paths (vercel/turbo#449)
this allows multiple versions of modules to co-exist in one chunking context e. g. RSC layer and SSR layer
2022-10-02 06:30:57 +00:00
Leah
2ffcbe38e0 simpler runtime inclusion (vercel/turbo#438) 2022-09-30 18:03:27 +00:00
Tobias Koppers
b9638d91d5 change WrapperAsset to some more general purpose VirtualAsset (vercel/turbo#447)
Wrapper is a bit of a weird name since it's not actually wrapping the existing asset. Actually it's just a virtual asset with a path below the existing path.
2022-09-30 11:58:35 +00:00
Tobias Koppers
1a5f27bd88 fixes some nitpicks (vercel/turbo#448) 2022-09-30 11:52:36 +00:00
Tobias Koppers
797bbc1a2f Refactor nodejs rendering to be more isolated and reusable (vercel/turbo#428)
(Prerequirement for `app` support)
2022-09-30 08:13:37 +00:00
Leah
9fd9163096 use embed macro (vercel/turbo#437) 2022-09-29 21:36:48 +00:00
Leah
fb76744de1 fix asset paths in server rendered pages (vercel/turbo#436)
Currently, when a server rendered page imports an asset (e.g. an image) there's a hydration mismatch
2022-09-29 17:14:15 +00:00