### What?
WEB-1150.
This PR is an attempt to upload next.js's test results into datadog test trace to track its status.
Originally it tried to use automatic trace injection (dd-trace/ci/init). However, due to some of custom environments / setup we use it was not compatible out of the box. Instead, this PR injects a new test reportert to generate junit report, then upload it at once at the end of the testing pipeline.
The reporter is configured to run when necessary variables set, local run should not be affected.
One thing to note is this report will not count retry results, as it'll create duplicated test entry with multiple results since we runts jest per individual test. This'll allow to detect flaky test easier, but also it means we'll get bit of skewed test results compare to the real world as first failure will be accounted as test fail immediately.
### What?
Another attempt to https://github.com/vercel/next.js/pull/50619 and WEB-1150, trying to apply setup guard more throughly.
I still do not know why original PR passed CI but fails to subsequent PRs after merge, but hope this could be a right guard to prevent unexpected failures.
### What?
This PR attempts to enable datadog trace integrations to the next.js integration tests if env is configured. With this, datadog can observe each test suite's results and detect some meaningful information (i.e flaky) for us.
However, I wasn't able to verify this works with next.js repo since for some reason CI worker does not pick up the api key in the env (https://vercel.slack.com/archives/C04KC8A53T7/p1685597124894539). Still this won't affect existing workflow, and once enabled I can test it over vercel/turbo repo instead.
Partially resolved WEB-1150.
This breaks up some of our longest running tests which allows more
parallelizing of the tests. This also moves turbopack tests filtering
back to an allow list as it is running a lot of unrelated tests
currently which isn't ideal. We should only be running against tests
that are explicitly testing turbopack e.g. build tests should not be
duplicated in the turbopack group.
```sh
test/integration/css/test/group-1.test.js: 762.655
test/integration/edge-runtime-module-errors/test/index.test.js: 695.309
test/integration/css/test/group-2.test.js: 671.848
test/integration/i18n-support/test/index.test.js: 518.173
test/integration/scss-modules/test/index.test.js: 451.704
test/integration/css-features/test/index.test.js: 417.318
test/integration/css-modules/test/index.test.js: 403.405
test/integration/eslint/test/index.test.js: 381.563
test/integration/500-page/test/index.test.js: 371.134
test/integration/telemetry/test/index.test.js: 367.691
test/development/acceptance-app/ReactRefreshLogBox.test.ts: 335.878
test/integration/create-next-app/templates.test.ts: 334.01
test/integration/scss/test/group-2.test.js: 327.255
test/integration/scss/test/group-1.test.js: 318.574
test/integration/edge-runtime-configurable-guards/test/index.test.js: 313.834
test/e2e/instrumentation-hook/instrumentation-hook.test.ts: 294.618
test/development/acceptance-app/error-recovery.test.ts: 283.355
test/e2e/app-dir/app/vercel-speed-insights.test.ts: 278.242
test/integration/create-next-app/index.test.ts: 272.442
```
This adds new `build and test` and `build and deploy` workflows in favor
of the existing massive `build, test, and deploy` workflow. Since the
new workflows will use `pull_request_target` this waits to remove the
existing workflow until the new one is tested.
While testing this new workflow flakey behavior in tests have also been
addressed. Along with the new workflow we will also be leveraging new
runners which allow us to run tests against the production binary of
`next-swc` so this avoids slight differences in tests we've seen due to
running against the dev binary.
Furthermore we will have a new flow for allowing workflow runs on PRs
from external forks which will either require a comment be checking a
box approving the run after each change or a label added by the team.
The new flow also no longer relies on `actions/cache` or similar which
have proven to be pretty unreliable.
Tests runs with the new workflow can be seen here
https://github.com/vercel/next.js/actions/runs/5100673508/jobs/9169416949
<!-- Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
## For Contributors
### Improving Documentation or adding/fixing Examples
- The "examples guidelines" are followed from our contributing doc
https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md
- Make sure the linting passes by running `pnpm build && pnpm lint`. See
https://github.com/vercel/next.js/blob/canary/contributing/repository/linting.md
### Fixing a bug
- Related issues linked using `fixes #number`
- Tests added. See:
https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md
### Adding a feature
- Implements an existing feature request or RFC. Make sure the feature
request has been accepted for implementation before opening a PR. (A
discussion must be opened, see
https://github.com/vercel/next.js/discussions/new?category=ideas)
- Related issues/discussions are linked using `fixes #number`
- e2e tests added
(https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Documentation added
- Telemetry added. In case of a feature if it's used or not.
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md
## For Maintainers
- Minimal description (aim for explaining to someone not on the team to
understand the PR)
- When linking to a Slack thread, you might want to share details of the
conclusion
- Link both the Linear (Fixes NEXT-xxx) and the GitHub issues
- Add review comments if necessary to explain to the reviewer the logic
behind a change
### Why?
### How?
Closes NEXT-
Fixes #
-->
### What?
WEB-1086.
This PR flips the way running next.js integration test with Turbopack,
instead of using allowlist to run selectively enabled test now uses
denylists. The immediate effect is now any new test being added (as a
new file) will be executed with Turbopack by default, and should be
manually excluded if it fails.
Still, the number of tests we run remains same.
<!-- Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
## For Contributors
### Improving Documentation or adding/fixing Examples
- The "examples guidelines" are followed from our contributing doc
https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md
- Make sure the linting passes by running `pnpm build && pnpm lint`. See
https://github.com/vercel/next.js/blob/canary/contributing/repository/linting.md
### Fixing a bug
- Related issues linked using `fixes #number`
- Tests added. See:
https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md
### Adding a feature
- Implements an existing feature request or RFC. Make sure the feature
request has been accepted for implementation before opening a PR. (A
discussion must be opened, see
https://github.com/vercel/next.js/discussions/new?category=ideas)
- Related issues/discussions are linked using `fixes #number`
- e2e tests added
(https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Documentation added
- Telemetry added. In case of a feature if it's used or not.
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md
## For Maintainers
- Minimal description (aim for explaining to someone not on the team to
understand the PR)
- When linking to a Slack thread, you might want to share details of the
conclusion
- Link both the Linear (Fixes NEXT-xxx) and the GitHub issues
- Add review comments if necessary to explain to the reviewer the logic
behind a change
### What?
### Why?
### How?
Closes NEXT-
Fixes #
-->
This PR adjusts manifests for the next.js test with Turbopack, as I
found upstream test keep changing and need to sync its latests state
into the manifest.
Manifest is now .js file contains 2 arrays, one for the enabled, and
others for the disabled. Disabled doesn't mean it's always failing
though.
When our `run-tests` util is used we can safely only create one instance
of the package tarballs and re-use for the tests and then cleaning up at
the end.
Ports changes from testing new workflow
0ceae76bf4
We want to make sure that our examples are not breaking.
We don't want to be slowed down by broken 3rd party packages, but we
need to ensure that examples covering next.js features are always green.
Added as a standalone workflow that doesn't parallelize. It will just
run on a cron schedule, so we can check for current status. We can add
Slack ping later if we need to.
The workflow tests just that our examples can build, so mostly TS
issues. We could definitely follow up with actual tests, but that is not
as low-hanging.
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Co-authored-by: Tim Neutkens <tim@timneutkens.nl>
<!--
Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
-->
- closes WEB-600
- partially resolves WEB-544
This PR applies minor ergonomics changes to the test runner. First,
allows to emit successful test reports if continue_on_error is enabled:
this allows to track total test stats with --turbo runs. Secondly allows
to specify custom timeouts for the e2e - as written in comment otherwise
it can exceed total 6 hours of job limit due to having lots of
timeout-related failing tests.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the
feature request has been accepted for implementation before opening a
PR.
- [ ] Related issues linked using `fixes #number`
- [ ]
[e2e](https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs)
tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm build && pnpm lint`
- [ ] The "examples guidelines" are followed from [our contributing
doc](https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md)
<!--
Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
-->
This is improved followup for
https://github.com/vercel/next.js/pull/45914, I realized I applied retry
count logic only for the teardown, not for the actual execution. PR
changes whole retrycount if predicate matches, also changes minor
ergonomics for the turbopack output with custom binary.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the
feature request has been accepted for implementation before opening a
PR.
- [ ] Related issues linked using `fixes #number`
- [ ]
[e2e](https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs)
tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm build && pnpm lint`
- [ ] The "examples guidelines" are followed from [our contributing
doc](https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md)
---------
<!--
Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
-->
Closes WEB-592.
This PR adds a naive checker named `skipRetryTestManifest`, which can be
loaded by env variable. It expects a formed json includes single array -
if it is available and matchs to the path for the failed test, it'll
skip retrying.
Primarily this is supposed to be used in conjunction with
`NEXT_TEST_CONTINUE_ON_ERROR` - when running all of the tests with
turbopack regardless of failure, there are certain set of tests
`expected` to fail currently but it keeps retrying to increase whole
test execution time. Having a known set of test to skip retry
potentially reduce those time.
Co-authored-by: JJ Kasper <jj@jjsweb.site>
<!--
Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
-->
Partially resolves WEB-544.
This PR adds a new detection to the external env variable
`NEXT_TEST_CONTINUE_ON_ERROR`. If those variable is set, even if a suite
of test fails test runner will keep executing remaining tests.
Turbopack (vercel/turbo) runs Next.js integration tests to track its
feature parity to the existing Next.js features and currently it is
expected to fail on certain test: however, we still want to execute all
of the tests to track how many tests are passing / fails.
Once this change lands turbopack need further verification to see if all
of tests are actually running.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the
feature request has been accepted for implementation before opening a
PR.
- [ ] Related issues linked using `fixes #number`
- [ ]
[e2e](https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs)
tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm build && pnpm lint`
- [ ] The "examples guidelines" are followed from [our contributing
doc](https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md)
Fixes handling in isolated tests for windows and adds initial setup to run the main `app-dir` test suite. Also adds retrying when fetching test timings fails due to rate limiting.
Closes: https://github.com/vercel/next.js/pull/44331
As noticed in
https://github.com/vercel/next.js/pull/42966#pullrequestreview-1181705091
we weren't resetting tests correctly on retry causing an incorrect pass.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have a helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the
feature request has been accepted for implementation before opening a
PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have a helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm build && pnpm lint`
- [ ] The "examples guidelines" are followed from [our contributing
doc](https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md)
<!--
Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change that you're making:
-->
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have a helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the
feature request has been accepted for implementation before opening a
PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have a helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm lint`
- [ ] The "examples guidelines" are followed from [our contributing
doc](https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md)
@ijjk moving this here.
Co-authored-by: JJ Kasper <jj@jjsweb.site>
Merge e2e tests customized `node_modules` with installed `node_modules`,
to let you debug easily locally without moving folder between
`node_modules` and `node_modules_bak`
Also add `optoutServerComponentsBundle` to config schema.
- Updates the React 18 test suite to the latest React version.
- Upgrade `react-refresh` module
Fixes#35518Fixes#35703
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
Co-authored-by: Jiachi Liu <4800338+huozhi@users.noreply.github.com>
* Update yarn PnP tests and disable swc file reading for PnP
* update job
* Update test
* add env variable
* update destory
* test one
* bump timeout
* update pnp install command
* only run pnp test
* add more logs
* handle exit signal
* dont inherit stdio for install
* update server start
* re-add test type
* add build log
* additional logging
* update build command
* remove separate timeout
* update install command
* install separate for better time info
* add cache pre-warming
* update yarn config
* enable other pnp tests
* Separate out tests
* fix-lint
* update path
* update test concurrency for isolated tests
* update retries
* Revert "update test concurrency for isolated tests"
This reverts commit 3a6e924df8ec61d55d3ee8a58d24cd50f0141195.
* re-enable production tests
* apply suggestions
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
This ensures we trim test output that will exceed the amount that can be shown in GitHub actions so that we can still see which tests failed instead of un-helpful test logs.
* Move unit tests to one folder
* Migrate unit tests to TypeScript
* add test types to lint
* Ensure ts(x) tests are run with util
* Add tsx extension to jest config
* bump
* Update polling env var for tests in CI
* Update polling value
* always use polling
* Revert "always use polling"
This reverts commit 281875de921874592a6755d67acaad7441c06e22.
* test disabling tcp/udp offloading
* fix quotes
This updates to only output a test's logs when it fails to reduce the noise in the CI's logs to allow easier investigating a failure. This also updates azure to leverage the `run-tests` script when testing ie11 to allow retrying similar to our other tests.