The PR follows #56536 and #56491, replacing `fs-extra` usages inside the `scripts/` folder.
Note that the `copy` and `move` haven't been replaced yet. Currently, there is no better recursive copy (lightweight, promise-based, Node.js built-in `copyFile` API-based, support the `filter` option) library alternative available on npm, and Node.js built-in `fs.rename` doesn't support `overwrite`.
The PR also replaces many async fs API usage with their sync versions.
cc @wbinnssmith
### What?
instead of include known passing tests
also updates the update test manifest
### Why?
Newly added test cases should always pass turbopack (or at least you would need to manually opt-out of it.
### How?
Uses a exclude list of tests instead of an allow list
Closes WEB-1752
### What?
A follow up to https://github.com/vercel/next.js/pull/55975, this will
list every flakey test in the turbopack suite so that we can run the
remaining passing tests.
### Why?
More tests is more better.
### How?
Trial and error. Gonna let CI tell me what's flakey with a few runs.
Closes WEB-1651
### What?
Updates `run-test.js` to allow running individual test cases inside a test file.
### Why?
So that we can dramatically increase Turbopack's test coverage. Turbopack has implemented most (but not all) necessary features from the webpack bundles. But a single failing test case would prevent us from running any case inside a test file. With case filtering, we're able to run the cases we know will pass and prevent regressions on those.
### How?
Case filtering is only exposed via the `NEXT_EXTERNAL_TESTS_FILTERS` ENV, which points to a JSON file containing a map of test files with the test cases inside those files that are known to pass.
The known-passing test cases can be updated after Turbopack's daily integration test run by running `test/build-turbopack-tests-manifest.js`, which will update the `test/turbopack-tests-manifest.json` manifest.
Closes WEB-1640
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
This runs unit tests separate from the `test-dev` task to make it a bit more obvious these are running, and allow for an easy way to add any unit-test specific behavior in the future
[slack x-ref](https://vercel.slack.com/archives/C04DUD7EB1B/p1695150908857269)
### What
Pairing PR to https://github.com/vercel/turbo/pull/5714, which I believe last piece to reenable correct datadog test reports for daily next.js integration + turbopack.
1. Creates a unique suite name to avoid collision / merging across different env / grouped test run
2. revert suitename template (which is for _suites_, not _suite_ for top level) as 1 resolves uniqueness
### What?
* fixes problems in Next.rs API introduced by #52846
* adds test infrastructure for experimental turbo testing
* adds two test cases to verify the infrastructure
* add grouping of output logs in run-tests
* simplify template loading
### Why?
### How?
This relocates the regex patterns that were being used for `test-dev`, `test-prod`, and `test-integration` actions to be part of `run-tests`. Also fixes e2e tests since they were expected to be found in `e2e` rather than `test/e2e`, and updated it exit with a failure code in case of no tests being found
Co-authored-by: JJ Kasper <22380829+ijjk@users.noreply.github.com>
Colocated unit tests (such as ones in `packages/next` and `packages/font`) weren't running in CI since `run-tests` marks the glob cwd as `<root>/tests`. This modifies the working directory to be the root so the new expanded test pattern will pick up files outside of `test/`.
Several of these tests were failing so there are updates in here to fix them. Specifically:
- Source Sans Pro font was renamed to Source Sans 3
- `fillCacheWithDataProperty` test was hitting the `bailOptimistic` code path
- `resolve-metadata` had an invalid assertion (`rel: icon` gets added as part of `IconsMetadata`)
- `resolve-opengraph` wasn't handling undefined images
- `server-patch-reducer` now use inline snapshots as one was failing since it now has a prefetchCache
### What?
WEB-1150.
This PR is an attempt to upload next.js's test results into datadog test trace to track its status.
Originally it tried to use automatic trace injection (dd-trace/ci/init). However, due to some of custom environments / setup we use it was not compatible out of the box. Instead, this PR injects a new test reportert to generate junit report, then upload it at once at the end of the testing pipeline.
The reporter is configured to run when necessary variables set, local run should not be affected.
One thing to note is this report will not count retry results, as it'll create duplicated test entry with multiple results since we runts jest per individual test. This'll allow to detect flaky test easier, but also it means we'll get bit of skewed test results compare to the real world as first failure will be accounted as test fail immediately.
### What?
Another attempt to https://github.com/vercel/next.js/pull/50619 and WEB-1150, trying to apply setup guard more throughly.
I still do not know why original PR passed CI but fails to subsequent PRs after merge, but hope this could be a right guard to prevent unexpected failures.
### What?
This PR attempts to enable datadog trace integrations to the next.js integration tests if env is configured. With this, datadog can observe each test suite's results and detect some meaningful information (i.e flaky) for us.
However, I wasn't able to verify this works with next.js repo since for some reason CI worker does not pick up the api key in the env (https://vercel.slack.com/archives/C04KC8A53T7/p1685597124894539). Still this won't affect existing workflow, and once enabled I can test it over vercel/turbo repo instead.
Partially resolved WEB-1150.
This breaks up some of our longest running tests which allows more
parallelizing of the tests. This also moves turbopack tests filtering
back to an allow list as it is running a lot of unrelated tests
currently which isn't ideal. We should only be running against tests
that are explicitly testing turbopack e.g. build tests should not be
duplicated in the turbopack group.
```sh
test/integration/css/test/group-1.test.js: 762.655
test/integration/edge-runtime-module-errors/test/index.test.js: 695.309
test/integration/css/test/group-2.test.js: 671.848
test/integration/i18n-support/test/index.test.js: 518.173
test/integration/scss-modules/test/index.test.js: 451.704
test/integration/css-features/test/index.test.js: 417.318
test/integration/css-modules/test/index.test.js: 403.405
test/integration/eslint/test/index.test.js: 381.563
test/integration/500-page/test/index.test.js: 371.134
test/integration/telemetry/test/index.test.js: 367.691
test/development/acceptance-app/ReactRefreshLogBox.test.ts: 335.878
test/integration/create-next-app/templates.test.ts: 334.01
test/integration/scss/test/group-2.test.js: 327.255
test/integration/scss/test/group-1.test.js: 318.574
test/integration/edge-runtime-configurable-guards/test/index.test.js: 313.834
test/e2e/instrumentation-hook/instrumentation-hook.test.ts: 294.618
test/development/acceptance-app/error-recovery.test.ts: 283.355
test/e2e/app-dir/app/vercel-speed-insights.test.ts: 278.242
test/integration/create-next-app/index.test.ts: 272.442
```
This adds new `build and test` and `build and deploy` workflows in favor
of the existing massive `build, test, and deploy` workflow. Since the
new workflows will use `pull_request_target` this waits to remove the
existing workflow until the new one is tested.
While testing this new workflow flakey behavior in tests have also been
addressed. Along with the new workflow we will also be leveraging new
runners which allow us to run tests against the production binary of
`next-swc` so this avoids slight differences in tests we've seen due to
running against the dev binary.
Furthermore we will have a new flow for allowing workflow runs on PRs
from external forks which will either require a comment be checking a
box approving the run after each change or a label added by the team.
The new flow also no longer relies on `actions/cache` or similar which
have proven to be pretty unreliable.
Tests runs with the new workflow can be seen here
https://github.com/vercel/next.js/actions/runs/5100673508/jobs/9169416949
<!-- Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
## For Contributors
### Improving Documentation or adding/fixing Examples
- The "examples guidelines" are followed from our contributing doc
https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md
- Make sure the linting passes by running `pnpm build && pnpm lint`. See
https://github.com/vercel/next.js/blob/canary/contributing/repository/linting.md
### Fixing a bug
- Related issues linked using `fixes #number`
- Tests added. See:
https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md
### Adding a feature
- Implements an existing feature request or RFC. Make sure the feature
request has been accepted for implementation before opening a PR. (A
discussion must be opened, see
https://github.com/vercel/next.js/discussions/new?category=ideas)
- Related issues/discussions are linked using `fixes #number`
- e2e tests added
(https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Documentation added
- Telemetry added. In case of a feature if it's used or not.
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md
## For Maintainers
- Minimal description (aim for explaining to someone not on the team to
understand the PR)
- When linking to a Slack thread, you might want to share details of the
conclusion
- Link both the Linear (Fixes NEXT-xxx) and the GitHub issues
- Add review comments if necessary to explain to the reviewer the logic
behind a change
### Why?
### How?
Closes NEXT-
Fixes #
-->
### What?
WEB-1086.
This PR flips the way running next.js integration test with Turbopack,
instead of using allowlist to run selectively enabled test now uses
denylists. The immediate effect is now any new test being added (as a
new file) will be executed with Turbopack by default, and should be
manually excluded if it fails.
Still, the number of tests we run remains same.
<!-- Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
## For Contributors
### Improving Documentation or adding/fixing Examples
- The "examples guidelines" are followed from our contributing doc
https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md
- Make sure the linting passes by running `pnpm build && pnpm lint`. See
https://github.com/vercel/next.js/blob/canary/contributing/repository/linting.md
### Fixing a bug
- Related issues linked using `fixes #number`
- Tests added. See:
https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md
### Adding a feature
- Implements an existing feature request or RFC. Make sure the feature
request has been accepted for implementation before opening a PR. (A
discussion must be opened, see
https://github.com/vercel/next.js/discussions/new?category=ideas)
- Related issues/discussions are linked using `fixes #number`
- e2e tests added
(https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Documentation added
- Telemetry added. In case of a feature if it's used or not.
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md
## For Maintainers
- Minimal description (aim for explaining to someone not on the team to
understand the PR)
- When linking to a Slack thread, you might want to share details of the
conclusion
- Link both the Linear (Fixes NEXT-xxx) and the GitHub issues
- Add review comments if necessary to explain to the reviewer the logic
behind a change
### What?
### Why?
### How?
Closes NEXT-
Fixes #
-->
This PR adjusts manifests for the next.js test with Turbopack, as I
found upstream test keep changing and need to sync its latests state
into the manifest.
Manifest is now .js file contains 2 arrays, one for the enabled, and
others for the disabled. Disabled doesn't mean it's always failing
though.
When our `run-tests` util is used we can safely only create one instance
of the package tarballs and re-use for the tests and then cleaning up at
the end.
Ports changes from testing new workflow
0ceae76bf4
We want to make sure that our examples are not breaking.
We don't want to be slowed down by broken 3rd party packages, but we
need to ensure that examples covering next.js features are always green.
Added as a standalone workflow that doesn't parallelize. It will just
run on a cron schedule, so we can check for current status. We can add
Slack ping later if we need to.
The workflow tests just that our examples can build, so mostly TS
issues. We could definitely follow up with actual tests, but that is not
as low-hanging.
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Co-authored-by: Tim Neutkens <tim@timneutkens.nl>
<!--
Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
-->
- closes WEB-600
- partially resolves WEB-544
This PR applies minor ergonomics changes to the test runner. First,
allows to emit successful test reports if continue_on_error is enabled:
this allows to track total test stats with --turbo runs. Secondly allows
to specify custom timeouts for the e2e - as written in comment otherwise
it can exceed total 6 hours of job limit due to having lots of
timeout-related failing tests.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the
feature request has been accepted for implementation before opening a
PR.
- [ ] Related issues linked using `fixes #number`
- [ ]
[e2e](https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs)
tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm build && pnpm lint`
- [ ] The "examples guidelines" are followed from [our contributing
doc](https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md)
<!--
Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
-->
This is improved followup for
https://github.com/vercel/next.js/pull/45914, I realized I applied retry
count logic only for the teardown, not for the actual execution. PR
changes whole retrycount if predicate matches, also changes minor
ergonomics for the turbopack output with custom binary.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the
feature request has been accepted for implementation before opening a
PR.
- [ ] Related issues linked using `fixes #number`
- [ ]
[e2e](https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs)
tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm build && pnpm lint`
- [ ] The "examples guidelines" are followed from [our contributing
doc](https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md)
---------
<!--
Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
-->
Closes WEB-592.
This PR adds a naive checker named `skipRetryTestManifest`, which can be
loaded by env variable. It expects a formed json includes single array -
if it is available and matchs to the path for the failed test, it'll
skip retrying.
Primarily this is supposed to be used in conjunction with
`NEXT_TEST_CONTINUE_ON_ERROR` - when running all of the tests with
turbopack regardless of failure, there are certain set of tests
`expected` to fail currently but it keeps retrying to increase whole
test execution time. Having a known set of test to skip retry
potentially reduce those time.
Co-authored-by: JJ Kasper <jj@jjsweb.site>
<!--
Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
-->
Partially resolves WEB-544.
This PR adds a new detection to the external env variable
`NEXT_TEST_CONTINUE_ON_ERROR`. If those variable is set, even if a suite
of test fails test runner will keep executing remaining tests.
Turbopack (vercel/turbo) runs Next.js integration tests to track its
feature parity to the existing Next.js features and currently it is
expected to fail on certain test: however, we still want to execute all
of the tests to track how many tests are passing / fails.
Once this change lands turbopack need further verification to see if all
of tests are actually running.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the
feature request has been accepted for implementation before opening a
PR.
- [ ] Related issues linked using `fixes #number`
- [ ]
[e2e](https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs)
tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm build && pnpm lint`
- [ ] The "examples guidelines" are followed from [our contributing
doc](https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md)
Fixes handling in isolated tests for windows and adds initial setup to run the main `app-dir` test suite. Also adds retrying when fetching test timings fails due to rate limiting.
Closes: https://github.com/vercel/next.js/pull/44331
As noticed in
https://github.com/vercel/next.js/pull/42966#pullrequestreview-1181705091
we weren't resetting tests correctly on retry causing an incorrect pass.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have a helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the
feature request has been accepted for implementation before opening a
PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have a helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm build && pnpm lint`
- [ ] The "examples guidelines" are followed from [our contributing
doc](https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md)
<!--
Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change that you're making:
-->
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have a helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the
feature request has been accepted for implementation before opening a
PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have a helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm lint`
- [ ] The "examples guidelines" are followed from [our contributing
doc](https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md)
@ijjk moving this here.
Co-authored-by: JJ Kasper <jj@jjsweb.site>
Merge e2e tests customized `node_modules` with installed `node_modules`,
to let you debug easily locally without moving folder between
`node_modules` and `node_modules_bak`
Also add `optoutServerComponentsBundle` to config schema.
- Updates the React 18 test suite to the latest React version.
- Upgrade `react-refresh` module
Fixes#35518Fixes#35703
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
Co-authored-by: Jiachi Liu <4800338+huozhi@users.noreply.github.com>
