Since we reset the test project on every e2e CI run, deployment protection is automatically enabled by default.
This adds an option to the reset project workflow to disable deployment protection. Our test runners need to be able to hit these pages from an unauthenticated browser in order for the tests to work.
Verified tests are running properly in [this run](https://github.com/vercel/next.js/actions/runs/6971348806/job/18971225559) (fixing any failing tests themselves are out of scope for this PR; will evaluate once the run finishes)
Closes NEXT-1732
We have identical `resetProject` code used in `bench/vercel` and our e2e workflow action -- this updates the `resetProject` script to side-effects free (hence removing the env var) and shared between bench & e2e
Closes NEXT-1731
Fixing + doing some maintenance as I was using it for some things.
Changes:
- fixed a bug related to how React is injected
- added a progress tracker for the numbers of requests
- retry on 500
- added a param for crashing the lambda manually (useful for cold boots)
- added a param to skip build, useful if you want to retest the same variation
- don't crash when there's no data to compare
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have a helpful link attached, see [`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
- [ ] Related issues linked using `fixes #number`
- [ ] [e2e](https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs) tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have a helpful link attached, see [`contributing.md`](https://github.com/vercel/next.js/blob/canary/contributing.md)
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm build && pnpm lint`
- [ ] The "examples guidelines" are followed from [our contributing doc](https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md)
This PR adds the benchmarking script I've been using for #40251 to
measure the performance improvements that we make to the Edge SSR
runtime.
This tool:
- uploads two version of the benchmarking project to Vercel, one with
the latest canary and the one with your current local changes in dist
(don't forget to build!)
- runs some tests against the published url to measure TTFB
- displays a nice chart and table
What this doesn't do (yet):
- allow you to choose which URL to compare
- allow you to change the measured metric
- run a battery of differnet test
## Bug
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Errors have a helpful link attached, see `contributing.md`
## Feature
- [ ] Implements an existing feature request or RFC. Make sure the
feature request has been accepted for implementation before opening a
PR.
- [ ] Related issues linked using `fixes #number`
- [ ] Integration tests added
- [ ] Documentation added
- [ ] Telemetry added. In case of a feature if it's used or not.
- [ ] Errors have a helpful link attached, see `contributing.md`
https://user-images.githubusercontent.com/11064311/191270204-04447e20-5a40-43a9-bcda-b7eaeb3d270a.mov
## Documentation / Examples
- [ ] Make sure the linting passes by running `pnpm lint`
- [ ] The "examples guidelines" are followed from [our contributing
doc](https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
