Commit graph

120 commits

Author SHA1 Message Date
LongYinan
8c80ba4f78 Migrate to pnpm (vercel/turbo#336)
### turbo tracing in pnpm projects

How pnpm link packages:

![image](https://user-images.githubusercontent.com/3468483/191022172-962af095-9b44-4b69-8e2f-1e8d440d1c31.png)

After turbo tracing:

```
dist
├── crates
│   └── turbopack
│       └── tests
│           └── node-file-trace
│               ├── integration
│               └── node_modules
│                   └── better-sqlite3 -> ../../../../../node_modules/.pnpm/better-sqlite3@7.6.2/node_modules/better-sqlite3
└── node_modules
    └── .pnpm
        ├── better-sqlite3@7.6.2
        │   └── node_modules
        │       ├── better-sqlite3
        │       │   ├── build
        │       │   │   └── Release
        │       │   └── lib
        │       │       └── methods
        │       └── bindings -> ../../bindings@1.5.0/node_modules/bindings
        ├── bindings@1.5.0
        │   └── node_modules
        │       ├── bindings
        │       └── file-uri-to-path -> ../../file-uri-to-path@1.0.0/node_modules/file-uri-to-path
        └── file-uri-to-path@1.0.0
            └── node_modules
                └── file-uri-to-path
```

No more **global .pnpm store**. The directory link will follow the original path created by pnpm.
2022-10-05 16:42:55 +00:00
Tobias Koppers
a8b0fd4f4b add layer to ChunkingContext and include that in module id and chunk paths (vercel/turbo#449)
this allows multiple versions of modules to co-exist in one chunking context e. g. RSC layer and SSR layer
2022-10-02 06:30:57 +00:00
Leah
2ffcbe38e0 simpler runtime inclusion (vercel/turbo#438) 2022-09-30 18:03:27 +00:00
Tobias Koppers
b9638d91d5 change WrapperAsset to some more general purpose VirtualAsset (vercel/turbo#447)
Wrapper is a bit of a weird name since it's not actually wrapping the existing asset. Actually it's just a virtual asset with a path below the existing path.
2022-09-30 11:58:35 +00:00
Tobias Koppers
1a5f27bd88 fixes some nitpicks (vercel/turbo#448) 2022-09-30 11:52:36 +00:00
Tobias Koppers
797bbc1a2f Refactor nodejs rendering to be more isolated and reusable (vercel/turbo#428)
(Prerequirement for `app` support)
2022-09-30 08:13:37 +00:00
Leah
9fd9163096 use embed macro (vercel/turbo#437) 2022-09-29 21:36:48 +00:00
Leah
fb76744de1 fix asset paths in server rendered pages (vercel/turbo#436)
Currently, when a server rendered page imports an asset (e.g. an image) there's a hydration mismatch
2022-09-29 17:14:15 +00:00
Tobias Koppers
3ad7c51768 use embed_file (vercel/turbo#432) 2022-09-28 19:32:09 +00:00
Tobias Koppers
a3dd9915d7 Use ValueToString for AssetReference instead of description() (vercel/turbo#430)
(Prerequirement for `app` support)
2022-09-28 19:02:39 +00:00
LongYinan
402c64be00 Introduce AssetContent to handle symlink assets (vercel/turbo#406)
* Fix DiskFileSystem::read_link

* pnpm-like integration test

* Introduce AssetContent to handle symlink assets

* Fix read_link on Windows

* Run clippy fix

* Rename `path` to `target`

Co-authored-by: Justin Ridgewell <justin@ridgewell.name>

* Split Windows specified code

* Add comments about Redirect is only represent Directory

* Handle symlink while reading content

* Clippy fix

* Revert previous changes in FileSystemPathVc::get_type

* Fix Unix read_link

* cleanup

* handle symlink while resolving native bindings

* Make LinkContent::Link contains only target

* Add LinkType to represent link type

* Cleanup VersionedAsset

* Cleanup LinkType

* Normalize the LinkContent::target

* Comments

* Revert special case workaround for sharp on Windows

* comments

* node_native_binding follow file link

* Apply CR suggestion

Co-authored-by: Justin Ridgewell <justin@ridgewell.name>
2022-09-28 22:44:32 +08:00
Tobias Koppers
3263f5e5e6 replace json with serde_json (vercel/turbo#398)
get rid of `json` in favor of only `serde_json` for consistency and less dependencies
2022-09-28 09:02:40 +00:00
Justin Ridgewell
25ca6b2e2c Implement .env loading (vercel/turbo#409)
This implements `.env` loading by taking advantage of the [dotenvy](https://docs.rs/dotenvy/latest/dotenvy/) crate. Unfortunately, this isn't as featureful as the npm `dotenv`, lacking `FOO=${BAR:-default}` default support (this might be important), `FOO=${MAYBE_UNDEFINED:-${BAR-:default}}` nested fallbacks, and `` FOO=`backtick` `` support.

This is then converted into a series of assignments for the client code. This is run before any of the user code.

```js
const env = process.env;

env["FOO"] = "bar";
//...
```

- - -

TODO:
- [x] ~~Replace `process.env.FOO`/`process.env["FOO"]` during chunk compilation~~
  - using a runtime module speeds up HMR because the code doesn't need to be recompiled.
- [x] I don't actually know how to test server rendering, so I assume it works but haven't verified.
2022-09-27 20:33:45 +00:00
Leah
b7e9d4e3f0 more type safe next-dev builder (vercel/turbo#388) 2022-09-21 17:14:24 +00:00
Leah
620a6e0aaf small fixes / improvements (vercel/turbo#341)
Many small things I found

The most important is probably the typescript transform

The remaining bits should hopefully be self-explanatory from the commit messages
2022-09-21 15:47:23 +00:00
Leah
647a87f8f0 better ssr error message (vercel/turbo#387) 2022-09-20 18:57:49 +00:00
LongYinan
b9c36e7748 Align eslint and prettier rules with turborepo (vercel/turbo#397) 2022-09-20 07:33:15 +00:00
Tobias Koppers
a9b5ada56a fail-safe bindings code for SSR (vercel/turbo#391) 2022-09-19 19:11:14 +02:00
Tobias Koppers
1f7df032e2 benchmarks need more time and skip faster when failing (vercel/turbo#393) 2022-09-19 19:01:18 +02:00
Tobias Koppers
65be8a0804 handle exceptions during waiting for events (vercel/turbo#383) 2022-09-19 15:22:05 +00:00
Alex Kirszenberg
e35ad2f2dc Use React.memo in the test/bench app (vercel/turbo#384)
With React.memo:
```
bench_hmr_to_commit/Turbopack CSR/30000 modules
                        time:   [50.608 ms 51.659 ms 52.553 ms]
```

Without React.memo:
```
bench_hmr_to_commit/Turbopack CSR/30000 modules
                        time:   [853.47 ms 1.0191 s 1.1873 s]
                        change: [+1543.4% +1872.7% +2207.8%] (p = 0.00 < 0.05)
                        Performance has regressed.
```

Since we're only ever editing the top-level triangle in our HMR benchmarks, we're incurring the time it takes for React to re-render the whole tree, which is a function of the number of components in said tree. By using `React.memo`, we can skip updating children components during HMR.
2022-09-19 15:00:22 +00:00
Tobias Koppers
00b7a92a39 measure HMR to eval and HMR to commit independent from each other (vercel/turbo#382)
HMR to commit include react-refresh time to apply the update in the react component tree
2022-09-19 14:48:18 +02:00
Tobias Koppers
6d1a83384a Benchmark summarize script, workflow and stability (vercel/turbo#381) 2022-09-19 09:24:55 +00:00
Tobias Koppers
36c3d2d80f SourceMap fixes and improvements (vercel/turbo#374)
moves the logic of creating SourceMap assets into the asset reference. This avoids depending on the chunk items in the references() method directly. It also avoids calling CodeVc::source_map() until the source map is read

avoid circular dependency in call graph

It also avoids checking `has_source_map()` and just inserts potential Source Maps assets for every chunk item. Checking `has_source_map()` seems unnecessary work to do for all chunk items, when we can just send an empty source map.

only expose SourceMaps for chunk items when HMR is enabled
2022-09-16 15:08:00 +00:00
Tobias Koppers
7795fafdc7 add typescript support (vercel/turbo#370)
picked typescript transform from https://github.com/vercel/turbo-tooling/pull/341

add resolve options context as global resolve config

enable typescript only for next-dev

move emulating logic from environment to resolve options context

Co-authored-by: Leah <8845940+ForsakenHarmony@users.noreply.github.com>
2022-09-16 09:22:37 +00:00
Tobias Koppers
86f1831b06 fix compare action (vercel/turbo#373) 2022-09-16 08:06:54 +02:00
Will Binns-Smith
cf1eadb5ac Implement styled-jsx with swc's styled_jsx (vercel/turbo#367)
This reverts commit ccf6c112d541f0a1a9f4e691f6220363550aaf01.
2022-09-15 13:51:45 -07:00
Tobias Koppers
22e996b90c ensure that HMR is really happening while measuring (vercel/turbo#365)
no need to test all module counts
2022-09-15 09:54:07 +00:00
Will Binns-Smith
b6d38cd4fd Revert "Implement styled-jsx with swc's styled_jsx" (vercel/turbo#366)
Revert "Implement styled-jsx with swc's styled_jsx (vercel/turbo#354)"

This reverts commit 255a38b07c98252bfe442ee363f07924da45288a.
2022-09-15 02:24:08 +02:00
Tobias Koppers
f4e28a07ef enable react refresh for vite (vercel/turbo#364)
It was doing a full refresh before
2022-09-15 00:04:49 +00:00
Will Binns-Smith
d4192f4bd5 Implement styled-jsx with swc's styled_jsx (vercel/turbo#354)
This implements support for styled-jsx in next-dev using swc's styled_jsx crate.

It's only applied in next-dev, and is only applied as a transform to app code, much like the react-refresh transform.

To do:
* [x] The transform doesn't seem to be applied. Pass the added test.

Test Plan: `cargo test -p next-dev --
test_crates_next_dev_tests_integration_turbopack_basic_styled_jsx
--nocapture`
2022-09-14 23:46:18 +00:00
Tobias Koppers
826998cf07 increase benchmark timeouts to be able to benchmark larger module counts (vercel/turbo#362) 2022-09-14 23:17:14 +00:00
Alex Kirszenberg
50aa9cc261 Initial support for getStaticProps (vercel/turbo#345)
Remaining questions:
* Should we have some static analysis for `getStaticProps` instead of looking into exports at runtime?
* For now, the output of `getStaticProps` (if defined) will always trump the value passed in as `data`. If we consider `data` to be the cached output of `getStaticProps` (in the future, as this is not yet implemented), this logic should be adapted.
2022-09-14 21:31:14 +00:00
Will Binns-Smith
eb93985d95 Benchmarks: allow simultaneously installing multiple packages from npm (vercel/turbo#339)
Previously, we ran multiple `npm install` operations in serial using multiple calls to `install_from_npm`. Instead, this allows us to express dependencies all at once as a single command to the npm cli, which should reduce the time we spend installing from npm and updating package.json.

Test Plan: Manually confirmed that package.json was updated correctly.  `cargo bench`.
2022-09-12 12:34:26 -07:00
Will Binns-Smith
aaa640def6 Benchmark Webpack (vercel/turbo#338)
This adds webpack 5 to the benchmark suite.

Test Plan: Manually confirmed package.json updates and webpack config written to temp dir correctly. `cargo bench`.
2022-09-12 11:09:26 -07:00
Will Binns-Smith
361ca5eca8 Split benchmark code into more modules (vercel/turbo#337)
This splits the benchmark code into more modules. Notes:

* ~Moved/left `get_bundlers()` and `get_module_counts()` to/in mod.rs. In particular, moving `get_bundlers()` to either bundle.rs or util.rs would lead to a circular dependency. These both also rely on env var configuration, so I figured this was a reasonable place for them.~
* The Bundler trait has its own module (not moved to util), since it's a top-level concern and not really a miscellaneous utility.
* Each bundler has its own module file.

Test Plan: `TURBOPACK_BENCH_BUNDLERS=all cargo test --benches -p next-dev -- --nocapture` and verify same output as before change.
2022-09-09 16:35:12 -07:00
Tobias Koppers
ecaf59b3e6 improve startup metric measurement (vercel/turbo#331)
This fixes the `startup` benchmark metric to show the time until first render, instead of being equal to hydration.

It also skips `hydration` for CSR bundlers since it's equal first render for them.

It now correctly measures the initial SSR as `startup`, so CLI start until page visible:
![image](https://user-images.githubusercontent.com/1365881/189152420-396b181b-5a3e-4902-881d-7c247fa43bd8.png)
2022-09-08 16:00:15 +00:00
Alex Kirszenberg
0a112ccb14 Fix Next.js 11 React version (vercel/turbo#333) 2022-09-08 15:18:53 +00:00
Tobias Koppers
378adc7112 cleanup nitpicks (vercel/turbo#330) 2022-09-08 14:54:09 +00:00
Tobias Koppers
6f312afedc allow to pass module counts list via env var (vercel/turbo#332)
add custom benchmark workflow

run large counts in CI
2022-09-08 16:46:19 +02:00
Alex Kirszenberg
a45572b642 Ensure pages close before app does (vercel/turbo#329) 2022-09-08 15:02:48 +02:00
Alex Kirszenberg
6f0fc67cab Stop processes gracefully when possible, add drop guard to pages (vercel/turbo#328) 2022-09-08 12:00:19 +00:00
Will Binns-Smith
dd217c71b9 Benchmark Parcel (vercel/turbo#322)
* Benchmark Parcel

* add Parcel to the CI benchmarks

* move some turbopack dependencies to the bundler as they conflict with other bundlers

Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
2022-09-08 13:08:25 +02:00
Will Binns-Smith
abcdd78623 Benchmark Next.js 11 and 12 (vercel/turbo#304)
This implements benchmark support for Next.js 12. Next.js (the tool) expects to be able to resolve from the `next` package in the cwd, so it must be installed alongside the other node_modules in the test.  `prepare` was added to the Bundler trait to handle this case.

Test Plan: `TURBOPACK_BENCH_ALL=all cargo bench -p next-dev`


Co-authored-by: Alex Kirszenberg <1621758+alexkirsz@users.noreply.github.com>
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
2022-09-08 05:00:32 +00:00
Tobias Koppers
512539821d fix runtime code for server rendered content source (vercel/turbo#315)
move transition into turbopack
2022-09-07 15:56:10 +02:00
Will Binns-Smith
12e5f19a84 Benchmark against Vite (vercel/turbo#299)
This adds a comparison against Vite to our benchmark suite, running the startup, change, and restart benchmarks.

Test Plan: `cargo bench`


Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
2022-09-07 00:09:06 +00:00
Tobias Koppers
b78ede6099 fix benchmark compare actions (vercel/turbo#318)
The benchmark github action didn't correctly copy and use the executables used in benchmarks via CARGO_BIN_EXE_*.

This adds a CARGO_BIN_EXE_* runtime override to the github action to allow to use correct executable.

It will show correct results for next-dev benchmarks too, e. g. here an example of reverting vercel/turbo#295 
![image](https://user-images.githubusercontent.com/1365881/188696769-d393cd45-b08f-4fd5-ad69-2d38a54bcd92.png)
2022-09-06 17:34:51 +00:00
Tobias Koppers
7df0c410fc improve HMR update performance (vercel/turbo#295)
* avoid cloning Strings and reading/calling many functions when building ecmascript chunk content
* introduce `ReadRef` to allow storing snapshot of a value in values
* use snapshot trees to allow caching more function calls and reads

Before:
```
updated in 35ms (18 tasks)
updated in 34ms (18 tasks)
updated in 37ms (18 tasks)
updated in 31ms (18 tasks)
updated in 40ms (19 tasks)
updated in 37ms (18 tasks)
updated in 37ms (18 tasks)
updated in 34ms (18 tasks)
updated in 35ms (18 tasks)
updated in 52ms (18 tasks)
```

After:
```
updated in 6.105ms (19 tasks)
updated in 5.279ms (18 tasks)
updated in 10.471ms (19 tasks)
updated in 6.863ms (18 tasks)
updated in 4.593ms (18 tasks)
updated in 4.173ms (18 tasks)
updated in 5.352ms (18 tasks)
updated in 10.69ms (18 tasks)
updated in 5.065ms (18 tasks)
updated in 6.309ms (19 tasks)
```

a 5x performance improvement
2022-09-06 14:02:07 +00:00
Will Binns-Smith
aab23b38da Parameterize devserver used in benchmarks (vercel/turbo#298)
This implements the basics of parameterizing the tool/devserver used in these tests. Following PRs will implement benchmarking of Vite, bun, Parcel, etc.

Test Plan: `cargo bench -p next-dev` and verify no change in performance.
2022-09-06 09:39:11 +00:00
Will Binns-Smith
7d79ce5029 Benchmark restart time for dev server (vercel/turbo#244)
This implements a benchmark of restarting the devserver after successfully starting it and shutting it down.

## Question/TODO: 

Since our goal is metrics that don't scale with project size, should we
assert that the small/medium benchmark results don't differ?

Test Plan: `cargo bench -p next-dev`
2022-09-06 08:30:08 +00:00