docs(turbopack): conslidate existing links (#62034)

### What?

This PR improves turbopack documentation, mostly collects existing
documents into a single place.

Also, adds a support to render mermaid diagram.

Closes PACK-2488

.eslintignore

@@ -38,3 +38,4 @@ bench/nested-deps-app-router/**
 packages/next-bundle-analyzer/index.d.ts
 examples/with-typescript-graphql/lib/gql/
 test/development/basic/hmr/components/parse-error.js
+packages/next-swc/docs/assets/**/*

.github/workflows/build_and_deploy.yml

@@ -490,7 +490,14 @@ jobs:
         uses: peaceiris/actions-mdbook@v1
         with:
           mdbook-version: 'latest'
+      - name: Install cargo-binstall
+        uses: cargo-bins/cargo-binstall@v1.6.3
+
       - uses: Swatinem/rust-cache@v2
+
+      - run: |
+          cargo binstall mdbook-mermaid --no-confirm
+
       - name: Deploy preview docs
         if: ${{ needs.build.outputs.isRelease != 'true' }}
         run: ./scripts/deploy-turbopack-docs.sh

.github/workflows/build_and_test.yml

@@ -148,7 +148,7 @@ jobs:
       needsRust: 'yes'
       skipInstallBuild: 'yes'
       skipNativeBuild: 'yes'
-      afterBuild: ./scripts/deploy-turbopack-docs.sh
+      afterBuild: cargo binstall mdbook-mermaid --no-confirm && ./scripts/deploy-turbopack-docs.sh
       stepName: 'rust-doc-check'
     secrets: inherit
 

.github/workflows/build_reusable.yml

@@ -114,6 +114,9 @@ jobs:
         with:
           mdbook-version: 'latest'
 
+      - name: Install cargo-binstall
+        uses: cargo-bins/cargo-binstall@v1.6.3
+
       - name: 'Install mold linker'
         if: ${{ inputs.mold == 'yes' }}
         run: |

.prettierignore

@@ -20,6 +20,7 @@ packages/next/src/bundles/webpack/packages/lazy-compilation-*.js
 packages/next-swc/crates/**/tests/**/output*
 packages/next-swc/crates/core/tests/loader/issue-32553/input.js
 packages/next-swc/native/**/*
+packages/next-swc/docs/assets/**/*
 
 packages/next-codemod/transforms/__testfixtures__/**
 packages/next-codemod/transforms/__tests__/**

packages/next-swc/docs/assets/mermaid-init.js

@@ -0,0 +1 @@
+mermaid.initialize({ startOnLoad: true })

packages/next-swc/docs/book.toml

@@ -4,3 +4,13 @@ language = "en"
 multilingual = false
 src = "src"
 title = "Turbopack"
+
+[preprocessor]
+
+[preprocessor.mermaid]
+command = "mdbook-mermaid"
+
+[output]
+
+[output.html]
+additional-js = ["assets/mermaid.min.js", "assets/mermaid-init.js"]

packages/next-swc/docs/src/SUMMARY.md

@@ -6,21 +6,25 @@
 
 - [Setup dev environment](setup.md)
 - [How to run tests](how_to_run_tests.md)
+- [Tracing Turbopack](tracing_turbopack.md)
 
 # How Turbopack works
 
 - [Architecture](architecture.md)
+  - [Turbo tasks](turbo_tasks.md)
+  - [Turbopack](turbopack.md)
 
 # Next.js integration
 
 - [Turbopack binding](turbopack_binding.md)
-
-## Custom transforms
-
-- [Next/dynamic](next_dynamic.md)
-- [React server components](rsc.md)
+- [Next.rs api](next_rs_api.md)
+- [Contexts](turbopack_next_contexts.md)
+- [Custom transforms](turbopack_custom_transforms.md)
+  - [Next/dynamic](next_dynamic.md)
+  - [React server components](rsc.md)
 
 # Appendix
 
 - [API references](api.md)
+- [Additional Links](links.md)
 - [Writing documentation](documentation.md)

packages/next-swc/docs/src/architecture.md

@@ -1 +1,183 @@
 # Architecture
+
+This is a high level overview of Turbopack's architecture. For a deeper understanding in specific topic, check out the following docs:
+
+- [turbo tasks](turbo_tasks.md)
+
+## Core Concepts
+
+Turbopack has three guiding principles:
+
+### Incremental
+
+- Avoid operations that require broad/complete graph traversal
+- Changes should only have local effects
+- Avoid depending on "global" information (information from the whole app)
+- Avoid "one to many" dependencies (depending on one piece of information from many sources)
+
+### Lazy
+
+- Avoid computing more information than needed for a given operation
+- Put extra indirection when needed
+
+### Extensible
+
+- Use traits
+- Avoid depending on concrete implementations (avoid casting)
+- Make behavior configurable by passing options
+
+## Layers
+
+Turbopack models the user's code as it travels through Turbopack in multiple ways, each of which can be thought of as a layer in the system below:
+
+```
+
+┌───────────────┬──────────────────────────────┐
+│  Source code  │                              │
+├───────────────┘                              │
+│                        What the user wrote as│
+│                              application code│
+└──────────────────────────────────────────────┘
+
+┌───────────────┬──────────────────────────────┐
+│    Modules    │                              │
+├───────────────┘                              │
+│           The compiler's understanding of the│
+│                              application code│
+└──────────────────────────────────────────────┘
+
+┌───────────────┬──────────────────────────────┐
+│ Output assets │                              │
+├───────────────┘                              │
+│    What the target environment understands as│
+│                        executable application│
+└──────────────────────────────────────────────┘
+
+```
+
+Each layer is implemented as a decorator/wrapper around the previous layer in top-down order. In contrast to tools like webpack, where whole {source,module,chunk} graphs are built in serial, Turbopack builds lazily by successively wrapping an asset in each layer and building only what's needed without blocking on everything in a particular phase.
+
+### Source code
+
+- This is the original, unmodified code from the user's application
+- Does **not** model references (the relationships between files like through `import`)
+
+### Modules
+
+- Result of parsing and transpiling source code
+- Includes references to other modules
+- If references are followed, can traverse a subgraph of the application's transpiled code and its recursive dependencies
+
+### Output asset
+
+- These are chunks
+- Result of transforming a module given an output format
+- Includes references to other output assets
+- Can be emitted to disk or served from the dev server
+
+```mermaid
+graph TD
+  subgraph Source code
+    src ---> b.src.js[ ]
+    src ---> util
+    util ---> d.src.css[ ]
+    util ---> c.src.js[ ]
+  end
+  subgraph Module graph
+    a.ts ---> b.js
+    a.ts ---> c.js
+    b.js ---> c.js
+    c.js ---> d.css
+    c.js ---> e.js[ ]
+    b.js ---> f.js[ ]
+
+    b.js -.-> b.src.js
+    d.css -.-> d.src.css
+  end
+  subgraph Output graph
+    main.js --> chunk.js
+    main.js --> chunk2.js[ ]
+
+    main.js -.-> a.ts
+    main.js -.-> b.js
+    main.js -.-> c.js
+  end
+```
+
+## Output format
+
+An output format changes how a module is run in a specific target environment. The output format is chosen based on target environment and the input module.
+
+```mermaid
+graph LR
+    Browser ---> css_chunking[CSS chunking]
+    Browser ---> static_asset[Static asset]
+    Browser ---> js_chunking[JS chunking]
+
+    Node.js ---> Rebasing
+
+    vercel_nodejs[Vercel Node.js] ---> nft.json
+
+    bundler[Bundler or module-native target] ---> Modules
+```
+
+| Target         | Input        | Output format                            |
+| -------------- | ------------ | ---------------------------------------- |
+| Script tags    | JS/TS        | JS Chunking                              |
+| Link tags      | CSS          | CSS Chunking                             |
+| Browser        | Static asset | Static Asset (no chunking)               |
+| Node.js        | JS           | Rebasing (node-file trace build mode)    |
+| Node.js@Vercel | JS           | nft.json file (node-file-trace annotate) |
+
+In the future, Turbopack should support producing modules as an output format, to be used by other bundlers or runtimes that natively support ESM modules.
+
+## Chunking
+
+Chunking is the process that decides which modules are placed into which bundles, and the relationship between these bundles.
+
+```mermaid
+
+graph LR
+module[Module] --->|place in a Chunk| chunk_item[Chunk Item]
+module_[Module] --->|wrap in Chunk| Chunk
+
+```
+
+In order to be placed into chunks (bundles), a module must be wrapped in a Chunk Item. Doing so re-generates the module code, e.g. using `__turbopack_require__`. This still strictly wraps the content of a single module, meaning it assumes that any dependencies must be available to be required.
+
+If a module is placeable in a chunk, it can be wrapped in a Chunk, which will traverse all of its references.
+
+## Mix and match
+
+Not only can Turbopack travel between these different layers, but it can do so across and between different environments.
+
+- It's possible to transitions between Output Formats
+  - For example, starting in ecmascript but referencing static assets with `new URL('./foo.png')`
+- Embed/Reference Output Assets in Modules
+  - e.g. embedding base64 data url of a chunk inside another chunk
+- SSR Page references client chunks for hydration
+  - embed urls to client-side chunks into server-generated html
+  - references output assets from a different chunking root from a different environment
+- Client Component referenced from Server Component
+- Client component wrapper (Server Component) references client chunks of a client component
+
+All of this is made possible by having a single graph that models all source, modules, and output assets across all different environments.
+
+## Crates
+
+Currently `turbopack` splits into `turbopack-core`, `turbopack-css`
+, `turbopack-ecmascript` and `turbopack` (facade).
+
+The `turbopack-*` crates (except for `turbopack-core`) add support for a certain
+type of asset.
+
+Each of them export a `ModuleAsset`, an implementation of `Asset`, which is able
+to extract references from this kind of asset (and in the future generate an
+optimized (either for dev or prod) version of it).
+
+We are currently missing:
+
+- `turbopack-node` (node native modules)
+- `turbopack-wasm` (WASM in node and on the web)
+- `turbopack-image` (images on the web)
+- probably more? (e.g. `turbopack-svg`).

packages/next-swc/docs/src/documentation.md

@@ -4,7 +4,7 @@ Turbopack has two main types of documentation. One is a description of the code
 
 ## Write a general overview document
 
-Files written in markdown format in `packages/next-swc/docs/src` can be read by [mdbook](https://rust-lang.github.io/mdBook/).(Like the page you're looking at now)
+Files written in markdown format in `packages/next-swc/docs/src` can be read by [mdbook](https://rust-lang.github.io/mdBook/)(Like the page you're looking at now). Our configuration includes [mermaid](https://mermaid.js.org/) support to include diagrams. See [archtecture](archtecture.md) how it looks like.
 
 ## Write a rustdoc comment
 
@@ -13,3 +13,21 @@ If you write a [documentation comment](https://doc.rust-lang.org/reference/comme
 ## Build the documentation
 
 The `scripts/deploy-turbopack-docs.sh` script bundles the mdbook and rustdoc and deploys them. You can use it locally as a build script because it won't run the deployment if it doesn't have a token. The script runs a rustdoc build followed by an mdbook build, and then copies the output of the rustdoc build under `mdbook/rustdoc`.
+
+To locally build the documentation, ensure these are installed
+
+- Rust compiler
+- [mdbook](https://github.com/rust-lang/mdBook)
+- [mdbook-mermaid](https://github.com/badboy/mdbook-mermaid)
+
+```
+./scripts/deploy-turbopack-docs.sh
+open ./target/mdbook/index.html
+```
+
+or
+
+```
+// note you can't check rustdoc links when running devserver
+mdbook serve --dest-dir $(pwd)/target/mdbook ./packages/next-swc/docs
+```

packages/next-swc/docs/src/links.md

@@ -0,0 +1,16 @@
+# Links
+
+The links below are additional materials such as conference recording. Some recordings are password protected. Ask in team channel to get access, or check the Notion page.
+
+- [Turbopack and Turbo Tasks Walk-Through](https://vercel.zoom.us/rec/share/QViRx-rjuTjVW185oneDmqHTRPR9JLiOcvBon8Y29FUxbaMVRZjkCGNjjI2v38XW._G0VG6law7mXMF_5?startTime=1664462085000)
+- [Vercel presents: Turborepo/Turbopack with Tobias Koppers](https://vercel.zoom.us/rec/share/jExo8Yds0sIbf3rdktzi3b5MLP7UntiFe07QPuIjihfBh5XD76Qfwca5Srgo6AvH.6TLKT2JDxG44bNgZ)
+- [Turbopack General Overview and Chunking](https://loom.com/share/933ae237baf94c4387eb0759754ef898)
+- [Next.js Turbo Introduction](https://vercel.zoom.us/rec/share/t6O17-yI9rdbA8pyDpTRmdhCRQfgCfLM78xPgwXu1vb0MNLtCo1HlhcrgcYKiB9G.bfImJuKlfdc5xM6t) or [Loom](https://mandrillapp.com/track/click/30773215/loom.com?p=eyJzIjoiRzVISXVQQVRmd0dRbXlEMU5yTjFCZGt0dXdZIiwidiI6MSwicCI6IntcInVcIjozMDc3MzIxNSxcInZcIjoxLFwidXJsXCI6XCJodHRwczpcXFwvXFxcL2xvb20uY29tXFxcL3NoYXJlXFxcL2M3NzI5NzdkY2IxZTRiMzE4NmNhNDlkNjA4OTdiZTk3XCIsXCJpZFwiOlwiYzk0NzFjZWZjYTM1NDM3MThlYzQ2NzQyZTlkN2I0ZTlcIixcInVybF9pZHNcIjpbXCI0YjgwYjlkMWVkN2JmZDVjZmM1YjhhMDg2ZWFhNTJhZDhkZTQ0YWMxXCJdfSJ9)
+- [Turbopack Next.js: How to use the Tracing (loom.com)](https://www.loom.com/share/3ddfa5c0e45f45208b08e9093996f4f2)
+- [Turbopack Long Term Caching (loom.com)](https://www.loom.com/share/8eeb72866cc74a39b915363d6dfd99a3)
+- [Side-effect-free Modules Optimization (loom.com)](https://www.loom.com/share/bdee0c5f47ee4d32b7fcf183392ccaa3)
+- [Turbopack Intro](https://vercel.zoom.us/rec/share/zYfVUBl8W-rOn96VLPmJGm6XwDsUCSE0rzmLcsxXAPIP8eF0nywYY_5Hqe-HWN1R.KU8XxcBamIUMU69b)
+
+- [GitNation: The Core of Turbopack Explained (Live Coding)](https://portal.gitnation.org/contents/the-core-of-turbopack-explained-live-coding)
+- [GitNation(Rethinking Bundling Strategies)](https://portal.gitnation.org/contents/rethinking-bundling-strategies)
+- [GitNation(Turbopack. Why? How? When? and the Vision...)](https://portal.gitnation.org/contents/turbopack-why-how-when-and-the-vision)

packages/next-swc/docs/src/next_rs_api.md

@@ -0,0 +1 @@
+# Next.rs api

packages/next-swc/docs/src/tracing_turbopack.md

@@ -0,0 +1 @@
+# Tracing Turbopack

packages/next-swc/docs/src/turbo_tasks.md

@@ -0,0 +1,29 @@
+# Turbo tasks
+
+## idea
+
+## concept
+
+## Tasks
+
+## Cells and Vcs
+
+## Passing by value (TaskInput)
+
+## Best practices
+
+### Singleton
+
+## turbo-tasks-fs, turbo-tasks-env
+
+## Advanced
+
+### Serialization
+
+### Registry, turbo-tasks-build, register
+
+### State
+
+### Garbage collection
+
+### Determinism

packages/next-swc/docs/src/turbopack.md

@@ -0,0 +1,56 @@
+# Turbopack
+
+## Sources
+
+### FileSource
+
+### VirtualSource
+
+## Modules
+
+### AssetContext
+
+### ModuleReference
+
+    #### Reference types
+
+## OuputAssets
+
+## Crate structure
+
+## Advanced
+
+### Resolving
+
+    #### ResolveOptions
+
+### Layers
+
+### Chunking
+
+    #### AvailablilityInfo
+    #### Chunk groups
+
+### Transitions
+
+### CompileTimeInfo
+
+### SourceMaps
+
+### Issues
+
+## Ecmascript
+
+### VarGraph
+
+### Linking
+
+## CSS
+
+## JSON
+
+## Assets
+
+## Images
+
+## Non ecma inputs (MDX)

packages/next-swc/docs/src/turbopack_custom_transforms.md

@@ -0,0 +1 @@
+# Custom transforms

packages/next-swc/docs/src/turbopack_next_contexts.md

@@ -0,0 +1 @@
+# Contexts