We want to make sure that our examples are not breaking.
We don't want to be slowed down by broken 3rd party packages, but we
need to ensure that examples covering next.js features are always green.
Added as a standalone workflow that doesn't parallelize. It will just
run on a cron schedule, so we can check for current status. We can add
Slack ping later if we need to.
The workflow tests just that our examples can build, so mostly TS
issues. We could definitely follow up with actual tests, but that is not
as low-hanging.
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Co-authored-by: Tim Neutkens <tim@timneutkens.nl>
<!--
Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
-->
- closes WEB-600
- partially resolves WEB-544
This PR applies minor ergonomics changes to the test runner. First,
allows to emit successful test reports if continue_on_error is enabled:
this allows to track total test stats with --turbo runs. Secondly allows
to specify custom timeouts for the e2e - as written in comment otherwise
it can exceed total 6 hours of job limit due to having lots of
timeout-related failing tests.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the
feature request has been accepted for implementation before opening a
PR.
- [ ] Related issues linked using `fixes #number`
- [ ]
[e2e](https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs)
tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm build && pnpm lint`
- [ ] The "examples guidelines" are followed from [our contributing
doc](https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md)
<!--
Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
-->
This is improved followup for
https://github.com/vercel/next.js/pull/45914, I realized I applied retry
count logic only for the teardown, not for the actual execution. PR
changes whole retrycount if predicate matches, also changes minor
ergonomics for the turbopack output with custom binary.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the
feature request has been accepted for implementation before opening a
PR.
- [ ] Related issues linked using `fixes #number`
- [ ]
[e2e](https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs)
tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm build && pnpm lint`
- [ ] The "examples guidelines" are followed from [our contributing
doc](https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md)
---------
<!--
Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
-->
Closes WEB-592.
This PR adds a naive checker named `skipRetryTestManifest`, which can be
loaded by env variable. It expects a formed json includes single array -
if it is available and matchs to the path for the failed test, it'll
skip retrying.
Primarily this is supposed to be used in conjunction with
`NEXT_TEST_CONTINUE_ON_ERROR` - when running all of the tests with
turbopack regardless of failure, there are certain set of tests
`expected` to fail currently but it keeps retrying to increase whole
test execution time. Having a known set of test to skip retry
potentially reduce those time.
Co-authored-by: JJ Kasper <jj@jjsweb.site>
<!--
Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:
-->
Partially resolves WEB-544.
This PR adds a new detection to the external env variable
`NEXT_TEST_CONTINUE_ON_ERROR`. If those variable is set, even if a suite
of test fails test runner will keep executing remaining tests.
Turbopack (vercel/turbo) runs Next.js integration tests to track its
feature parity to the existing Next.js features and currently it is
expected to fail on certain test: however, we still want to execute all
of the tests to track how many tests are passing / fails.
Once this change lands turbopack need further verification to see if all
of tests are actually running.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the
feature request has been accepted for implementation before opening a
PR.
- [ ] Related issues linked using `fixes #number`
- [ ]
[e2e](https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs)
tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have a helpful link attached, see
[`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm build && pnpm lint`
- [ ] The "examples guidelines" are followed from [our contributing
doc](https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md)
Fixes handling in isolated tests for windows and adds initial setup to run the main `app-dir` test suite. Also adds retrying when fetching test timings fails due to rate limiting.
Closes: https://github.com/vercel/next.js/pull/44331
As noticed in
https://github.com/vercel/next.js/pull/42966#pullrequestreview-1181705091
we weren't resetting tests correctly on retry causing an incorrect pass.
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have a helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the
feature request has been accepted for implementation before opening a
PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have a helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm build && pnpm lint`
- [ ] The "examples guidelines" are followed from [our contributing
doc](https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md)
<!--
Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change that you're making:
-->
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have a helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the
feature request has been accepted for implementation before opening a
PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have a helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm lint`
- [ ] The "examples guidelines" are followed from [our contributing
doc](https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md)
@ijjk moving this here.
Co-authored-by: JJ Kasper <jj@jjsweb.site>
Merge e2e tests customized `node_modules` with installed `node_modules`,
to let you debug easily locally without moving folder between
`node_modules` and `node_modules_bak`
Also add `optoutServerComponentsBundle` to config schema.
- Updates the React 18 test suite to the latest React version.
- Upgrade `react-refresh` module
Fixes#35518Fixes#35703
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
Co-authored-by: Jiachi Liu <4800338+huozhi@users.noreply.github.com>
* Update yarn PnP tests and disable swc file reading for PnP
* update job
* Update test
* add env variable
* update destory
* test one
* bump timeout
* update pnp install command
* only run pnp test
* add more logs
* handle exit signal
* dont inherit stdio for install
* update server start
* re-add test type
* add build log
* additional logging
* update build command
* remove separate timeout
* update install command
* install separate for better time info
* add cache pre-warming
* update yarn config
* enable other pnp tests
* Separate out tests
* fix-lint
* update path
* update test concurrency for isolated tests
* update retries
* Revert "update test concurrency for isolated tests"
This reverts commit 3a6e924df8ec61d55d3ee8a58d24cd50f0141195.
* re-enable production tests
* apply suggestions
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have helpful link attached, see `contributing.md`
## Documentation / Examples
- [ ] Make sure the linting passes by running `yarn lint`
This ensures we trim test output that will exceed the amount that can be shown in GitHub actions so that we can still see which tests failed instead of un-helpful test logs.
* Move unit tests to one folder
* Migrate unit tests to TypeScript
* add test types to lint
* Ensure ts(x) tests are run with util
* Add tsx extension to jest config
* bump
* Update polling env var for tests in CI
* Update polling value
* always use polling
* Revert "always use polling"
This reverts commit 281875de921874592a6755d67acaad7441c06e22.
* test disabling tcp/udp offloading
* fix quotes
This updates to only output a test's logs when it fails to reduce the noise in the CI's logs to allow easier investigating a failure. This also updates azure to leverage the `run-tests` script when testing ie11 to allow retrying similar to our other tests.
This PR upgrades `jest-worker` and `jest-cli` to the latest pre-release version, also removed `jest-circus` which is included in Jest by default. `jest-worker@next` includes a fix for memory leak that we need (https://github.com/facebook/jest/pull/11187).
Fixes#22925. This will also improve the OOM issue for `next dev` #15855.
jest retries seem to be masking test failures, like the `auto-export` one (and maybe others). I turned it off for now. The `auto-export` test fails when retries are turned off.
the output of this test failure was a bit unhelpful so I also improved it.
Many tests have anonymous page functions.
- Reduce jest retries to 2 for a total of 3 attempts
- Disable retries in `invalid-href` test. I noticed jest retries don't help when this test fails (see log output of https://github.com/vercel/next.js/runs/904147534).
