Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Playwright tests for Next 13 app using app router fail unless slowMo set fairly high #52850

Open
1 task done
AmyBlankenship opened this issue Jul 19, 2023 · 7 comments
Open
1 task done
Labels
bug Issue was opened via the bug report template.

Comments

@AmyBlankenship
Copy link

Verify canary release

  • I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
      Platform: win32
      Arch: x64
      Version: Windows 10 Home
    Binaries:
      Node: 16.14.2
      npm: N/A
      Yarn: N/A
      pnpm: N/A
    Relevant Packages:
      next: 13.4.11-canary.0
      eslint-config-next: N/A
      react: 18.2.0
      react-dom: 18.2.0
      typescript: 4.9.5
    Next.js Config:
      output: N/A

Which area(s) of Next.js are affected? (leave empty if unsure)

No response

Link to the code that reproduces this issue or a replay of the bug

https://github.com/AmyBlankenship/next-contacts-example/tree/feat/add-address

To Reproduce

  • Clone branch above
  • yarn install
  • npm run test:e2e

Note: if the tests fail, you need to run posttest:e2e manually

Describe the Bug

With the project as committed, the "adding address" test fails in chrome, firefox, and edge (note I disabled safari early on because I was getting failures only in safari and figured it was bc this is a windows machine--not sure if this is relevant).

If you add:

launchOptions: { slowMo: 1000, },

below line 35 of playwright.config, now chrome and edge pass.

If you change that to
launchOptions: { slowMo: 2000, },

all three browsers pass.

Expected Behavior

Should not need to set playwright to slowMo to make tests pass (and if that's necessary, it should be clearly documented). Note that I don't know how to see what the browser versions are in playwright, but I just installed the latest as part of trying to figure this out.

Which browser are you using? (if relevant)

No response

How are you deploying your application? (if relevant)

No response

@AmyBlankenship AmyBlankenship added the bug Issue was opened via the bug report template. label Jul 19, 2023
@matejtarca
Copy link

matejtarca commented Oct 19, 2023

Any news here? Currently experiencing the same issue with next 13.5.4 and playwright 1.39.0.
If the slowMo is not specified, the page suddenly changes in the middle of a test to a completely different one (happening in somewhat random places but usually after a mutation). I think it's because of the revalidation of the path in server actions 🤔

EDIT: I found a workaround by introducing a manual wait in Playwright right after accessing a new page. That way a global slowMo is not needed, and the tests pass. I want to point out that I am testing on a standalone production build, not a dev server, so I am unsure why this would be an issue.

@AmyBlankenship
Copy link
Author

I'm not working in Next at my current job, but thanks for the update

@sjconstantin
Copy link

I am seeing this issue also. Am am testing for this

await expect(this.page.getByText('Logged in as:')).toBeVisible();

This passes when testing against my app running in development mode. Running in production mode the test fails without slowMo. If I set slowMo to 2000 the test passes in production mode.

@pieterjanscheir
Copy link

Does anyone have any update or other workarounds for this issue?

@manavm1990
Copy link

I didn't know about slowMo until I came across this 🧵 . However, I have gone from using geByLabel to getByLocator and now there are at least fewer flake issues.

diff --git a/apps/web/e2e/auth.spec.ts b/apps/web/e2e/auth.spec.ts
index d7ca2fc..9f3a5a3 100644
--- a/apps/web/e2e/auth.spec.ts
+++ b/apps/web/e2e/auth.spec.ts
@@ -14,9 +14,7 @@ test("Login and out", async ({ page }) => {
   expect(process.env.TEST_PASSWORD).toBeDefined();
 
   await page.getByLabel("Email ✉️").fill(process.env.TEST_USERNAME!);
-  await page
-    .getByLabel("Password", { exact: true })
-    .fill(process.env.TEST_PASSWORD!);
+  await page.locator('input[name="password"]').fill(process.env.TEST_PASSWORD!);
 
   const loginButton = page.getByRole("button", { name: "Log In" });
   await loginButton.click();
@@ -37,7 +35,7 @@ test("Validation", async ({ page }) => {
   await usernameInput.fill(process.env.TEST_USERNAME!);
   await expect(alert).not.toBeVisible();
 
-  const passwordInput = page.getByLabel("Password", { exact: true });
+  const passwordInput = page.locator('input[name="password"]');
   await passwordInput.fill("a");
   await expect(alert).toContainText("Must be 8 or more characters long");
   await passwordInput.fill(process.env.TEST_PASSWORD!);

@jameslporter
Copy link

Have a next 14 app dir app and cannot get playwright to work hardly at all while running in dev mode. Switch to build & start and it works fine, passes every time and takes 40s +-4s. Overall dev mode is frustratingly slow and prod mode is crazy fast. Have some issues with turbo mode so I'm not using that. Using bun runtime and it seems stable just uses more memory than node did.

@ColeMurray

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Issue was opened via the bug report template.
Projects
None yet
Development

No branches or pull requests

7 participants