validate path lengths before writes and attempt to eagerly convert win paths #69141

arlyon · 2024-08-21T12:10:50Z

What?

Check and attempt to resolve potential file length issues on all platforms.

Why?

Each OS / fs has particular constraints on file name- and path lengths. On unix they are hard(ish) limits. On windows, there is a workaround.

We have a hit a case where we are producing files that exceed these limits.

How?

On windows, attempt to canonicalize. On macOS / linux, check these limits (using macOS as the common denominator) and provide meaningful errors.

Note: I have opted to explicitly enforce reasonable limits here rather than relying on the undefined limits of the host operating system. This means we may hit cases where we are overly restrictive, however this sets the scene for the next PR that ensures, given these limits, that we never end up producing paths in turbopack that would exceed our own restrictions, namely making sure that written assets never exceed 255 bytes.

arlyon · 2024-08-21T12:11:05Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @arlyon and the rest of your teammates on Graphite

ijjk · 2024-08-21T12:17:04Z

Failing test suites

Commit: 4b383dc

__NEXT_EXPERIMENTAL_PPR=true pnpm test-dev test/e2e/app-dir/rsc-basic/rsc-basic.test.ts (PPR)

app dir - rsc basics > should be able to call legacy react-dom/server APIs in client components
app dir - rsc basics > react@experimental > should opt into the react@experimental when enabling ppr
app dir - rsc basics > react@experimental > should opt into the react@experimental when enabling taint

Expand output

● app dir - rsc basics › should be able to call legacy react-dom/server APIs in client components

page.waitForSelector: Timeout 60000ms exceeded.
Call log:
  - waiting for locator('#markup')

  423 |     return this.chain(() => {
  424 |       return page
> 425 |         .waitForSelector(selector, { timeout, state: 'attached' })
      |          ^
  426 |         .then(async (el) => {
  427 |           // it seems selenium waits longer and tests rely on this behavior
  428 |           // so we wait for the load event fire before returning

  at waitForSelector (lib/browsers/playwright.ts:425:10)

● app dir - rsc basics › react@experimental › should opt into the react@experimental when enabling ppr

expect(received).toMatch(expected)

Expected substring: "-experimental-"
Received string:    ""

  669 |
  670 |         ssrPagesReactVersions.forEach((version) => {
> 671 |           expect(version).toMatch('-experimental-')
      |                           ^
  672 |         })
  673 |
  674 |         const browser = await next.browser('/app-react')

  at toMatch (e2e/app-dir/rsc-basic/rsc-basic.test.ts:671:27)
      at Array.forEach (<anonymous>)
  at forEach (e2e/app-dir/rsc-basic/rsc-basic.test.ts:670:31)

● app dir - rsc basics › react@experimental › should opt into the react@experimental when enabling taint

expect(received).toMatch(expected)

Expected substring: "-experimental-"
Received string:    ""

  669 |
  670 |         ssrPagesReactVersions.forEach((version) => {
> 671 |           expect(version).toMatch('-experimental-')
      |                           ^
  672 |         })
  673 |
  674 |         const browser = await next.browser('/app-react')

  at toMatch (e2e/app-dir/rsc-basic/rsc-basic.test.ts:671:27)
      at Array.forEach (<anonymous>)
  at forEach (e2e/app-dir/rsc-basic/rsc-basic.test.ts:670:31)

Read more about building and testing Next.js in contributing.md.

ijjk · 2024-08-21T12:22:11Z

Stats from current PR

Default Build (Increase detected ⚠️)

General

	vercel/next.js canary	vercel/next.js arlyon/attempt-fix-too-long-file	Change
buildDuration	17.7s	18.8s	⚠️ +1.2s
buildDurationCached	8.9s	7.8s	N/A
nodeModulesSize	356 MB	356 MB	N/A
nextStartRea..uration (ms)	436ms	436ms	✓

Client Bundles (main, webpack)

	vercel/next.js canary	vercel/next.js arlyon/attempt-fix-too-long-file	Change
5496-HASH.js gzip	37.9 kB	37.9 kB	N/A
5885.HASH.js gzip	169 B	169 B	✓
9560-HASH.js gzip	5.2 kB	5.19 kB	N/A
b9a693a6-HASH.js gzip	51.9 kB	51.9 kB	N/A
framework-HASH.js gzip	56.7 kB	56.7 kB	✓
main-app-HASH.js gzip	223 B	226 B	N/A
main-HASH.js gzip	32.5 kB	32.5 kB	N/A
webpack-HASH.js gzip	1.71 kB	1.71 kB	N/A
Overall change	56.9 kB	56.9 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js arlyon/attempt-fix-too-long-file	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js arlyon/attempt-fix-too-long-file	Change
_app-HASH.js gzip	193 B	193 B	✓
_error-HASH.js gzip	192 B	191 B	N/A
amp-HASH.js gzip	511 B	512 B	N/A
css-HASH.js gzip	343 B	343 B	✓
dynamic-HASH.js gzip	1.84 kB	1.84 kB	✓
edge-ssr-HASH.js gzip	266 B	265 B	N/A
head-HASH.js gzip	362 B	365 B	N/A
hooks-HASH.js gzip	391 B	392 B	N/A
image-HASH.js gzip	4.4 kB	4.39 kB	N/A
index-HASH.js gzip	268 B	268 B	✓
link-HASH.js gzip	2.81 kB	2.81 kB	✓
routerDirect..HASH.js gzip	327 B	328 B	N/A
script-HASH.js gzip	397 B	397 B	✓
withRouter-HASH.js gzip	323 B	324 B	N/A
1afbb74e6ecf..834.css gzip	106 B	106 B	✓
Overall change	5.96 kB	5.96 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js arlyon/attempt-fix-too-long-file	Change
_buildManifest.js gzip	751 B	744 B	N/A
Overall change	0 B	0 B	✓

Rendered Page Sizes

	vercel/next.js canary	vercel/next.js arlyon/attempt-fix-too-long-file	Change
index.html gzip	520 B	523 B	N/A
link.html gzip	535 B	537 B	N/A
withRouter.html gzip	517 B	518 B	N/A
Overall change	0 B	0 B	✓

Edge SSR bundle Size

	vercel/next.js canary	vercel/next.js arlyon/attempt-fix-too-long-file	Change
edge-ssr.js gzip	128 kB	127 kB	N/A
page.js gzip	173 kB	173 kB	N/A
Overall change	0 B	0 B	✓

Middleware size

	vercel/next.js canary	vercel/next.js arlyon/attempt-fix-too-long-file	Change
middleware-b..fest.js gzip	668 B	665 B	N/A
middleware-r..fest.js gzip	155 B	156 B	N/A
middleware.js gzip	29.7 kB	29.7 kB	N/A
edge-runtime..pack.js gzip	844 B	844 B	✓
Overall change	844 B	844 B	✓

Next Runtimes

	vercel/next.js canary	vercel/next.js arlyon/attempt-fix-too-long-file	Change
928-experime...dev.js gzip	322 B	322 B	✓
928.runtime.dev.js gzip	314 B	314 B	✓
app-page-exp...dev.js gzip	309 kB	308 kB	N/A
app-page-exp..prod.js gzip	122 kB	121 kB	N/A
app-page-tur..prod.js gzip	135 kB	133 kB	N/A
app-page-tur..prod.js gzip	130 kB	129 kB	N/A
app-page.run...dev.js gzip	298 kB	298 kB	N/A
app-page.run..prod.js gzip	118 kB	118 kB	N/A
app-route-ex...dev.js gzip	30.7 kB	30.7 kB	N/A
app-route-ex..prod.js gzip	20.7 kB	20.7 kB	N/A
app-route-tu..prod.js gzip	20.7 kB	20.7 kB	N/A
app-route-tu..prod.js gzip	20.5 kB	20.5 kB	N/A
app-route.ru...dev.js gzip	32.4 kB	32.3 kB	N/A
app-route.ru..prod.js gzip	20.5 kB	20.5 kB	N/A
pages-api-tu..prod.js gzip	9.62 kB	9.62 kB	✓
pages-api.ru...dev.js gzip	11.5 kB	11.5 kB	✓
pages-api.ru..prod.js gzip	9.61 kB	9.61 kB	✓
pages-turbo...prod.js gzip	21.6 kB	21.6 kB	✓
pages.runtim...dev.js gzip	27.5 kB	27.5 kB	✓
pages.runtim..prod.js gzip	21.6 kB	21.6 kB	✓
server.runti..prod.js gzip	57 kB	56.8 kB	N/A
Overall change	102 kB	102 kB	✓

build cache Overall increase ⚠️

	vercel/next.js canary	vercel/next.js arlyon/attempt-fix-too-long-file	Change
0.pack gzip	1.49 MB	1.49 MB	⚠️ +3.96 kB
index.pack gzip	127 kB	127 kB	⚠️ +232 B
Overall change	1.62 MB	1.62 MB	⚠️ +4.19 kB

Diff details

Diff for page.js

Diff too large to display

Diff for middleware.js

Diff too large to display

Diff for edge-ssr.js

Diff too large to display

Diff for image-HASH.js

@@ -1,7 +1,7 @@
 (self["webpackChunk_N_E"] = self["webpackChunk_N_E"] || []).push([
   [8358],
   {
-    /***/ 3428: /***/ (
+    /***/ 6324: /***/ (
       __unused_webpack_module,
       __unused_webpack_exports,
       __webpack_require__
@@ -9,7 +9,7 @@
       (window.__NEXT_P = window.__NEXT_P || []).push([
         "/image",
         function () {
-          return __webpack_require__(9007);
+          return __webpack_require__(4483);
         },
       ]);
       if (false) {
@@ -18,7 +18,7 @@
       /***/
     },
 
-    /***/ 8578: /***/ (module, exports, __webpack_require__) => {
+    /***/ 3156: /***/ (module, exports, __webpack_require__) => {
       "use strict";
       /* __next_internal_client_entry_do_not_use__  cjs */
       Object.defineProperty(exports, "__esModule", {
@@ -40,17 +40,17 @@
         __webpack_require__(671)
       );
       const _head = /*#__PURE__*/ _interop_require_default._(
-        __webpack_require__(5208)
+        __webpack_require__(9939)
       );
-      const _getimgprops = __webpack_require__(8580);
-      const _imageconfig = __webpack_require__(4446);
-      const _imageconfigcontextsharedruntime = __webpack_require__(1864);
-      const _warnonce = __webpack_require__(3787);
-      const _routercontextsharedruntime = __webpack_require__(6758);
+      const _getimgprops = __webpack_require__(4422);
+      const _imageconfig = __webpack_require__(1573);
+      const _imageconfigcontextsharedruntime = __webpack_require__(5345);
+      const _warnonce = __webpack_require__(447);
+      const _routercontextsharedruntime = __webpack_require__(9420);
       const _imageloader = /*#__PURE__*/ _interop_require_default._(
-        __webpack_require__(842)
+        __webpack_require__(2949)
       );
-      const _usemergedref = __webpack_require__(3435);
+      const _usemergedref = __webpack_require__(8863);
       // This is replaced by webpack define plugin
       const configEnv = {
         deviceSizes: [640, 750, 828, 1080, 1200, 1920, 2048, 3840],
@@ -371,7 +371,7 @@
       /***/
     },
 
-    /***/ 3435: /***/ (module, exports, __webpack_require__) => {
+    /***/ 8863: /***/ (module, exports, __webpack_require__) => {
       "use strict";
 
       Object.defineProperty(exports, "__esModule", {
@@ -440,7 +440,7 @@
       /***/
     },
 
-    /***/ 8580: /***/ (
+    /***/ 4422: /***/ (
       __unused_webpack_module,
       exports,
       __webpack_require__
@@ -456,9 +456,9 @@
           return getImgProps;
         },
       });
-      const _warnonce = __webpack_require__(3787);
-      const _imageblursvg = __webpack_require__(5112);
-      const _imageconfig = __webpack_require__(4446);
+      const _warnonce = __webpack_require__(447);
+      const _imageblursvg = __webpack_require__(7821);
+      const _imageconfig = __webpack_require__(1573);
       const VALID_LOADING_VALUES =
         /* unused pure expression or super */ null && [
           "lazy",
@@ -830,7 +830,7 @@
       /***/
     },
 
-    /***/ 5112: /***/ (__unused_webpack_module, exports) => {
+    /***/ 7821: /***/ (__unused_webpack_module, exports) => {
       "use strict";
       /**
        * A shared function, used on both client and server, to generate a SVG blur placeholder.
@@ -885,7 +885,7 @@
       /***/
     },
 
-    /***/ 7735: /***/ (
+    /***/ 4563: /***/ (
       __unused_webpack_module,
       exports,
       __webpack_require__
@@ -912,10 +912,10 @@
         },
       });
       const _interop_require_default = __webpack_require__(4345);
-      const _getimgprops = __webpack_require__(8580);
-      const _imagecomponent = __webpack_require__(8578);
+      const _getimgprops = __webpack_require__(4422);
+      const _imagecomponent = __webpack_require__(3156);
       const _imageloader = /*#__PURE__*/ _interop_require_default._(
-        __webpack_require__(842)
+        __webpack_require__(2949)
       );
       function getImageProps(imgProps) {
         const { props } = (0, _getimgprops.getImgProps)(imgProps, {
@@ -947,7 +947,7 @@
       /***/
     },
 
-    /***/ 842: /***/ (__unused_webpack_module, exports) => {
+    /***/ 2949: /***/ (__unused_webpack_module, exports) => {
       "use strict";
 
       Object.defineProperty(exports, "__esModule", {
@@ -982,7 +982,7 @@
       /***/
     },
 
-    /***/ 9007: /***/ (
+    /***/ 4483: /***/ (
       __unused_webpack_module,
       __webpack_exports__,
       __webpack_require__
@@ -999,8 +999,8 @@
 
       // EXTERNAL MODULE: ./node_modules/.pnpm/react@19.0.0-rc-49496d49-20240814/node_modules/react/jsx-runtime.js
       var jsx_runtime = __webpack_require__(8436);
-      // EXTERNAL MODULE: ./node_modules/.pnpm/next@file+..+main-repo+packages+next+next-packed.tgz_react-dom@19.0.0-rc-49496d49-20240814_re_mwmfw6pwgc47m57syyu4ylsqkm/node_modules/next/image.js
-      var next_image = __webpack_require__(5519);
+      // EXTERNAL MODULE: ./node_modules/.pnpm/next@file+..+diff-repo+packages+next+next-packed.tgz_react-dom@19.0.0-rc-49496d49-20240814_re_q22c6uapaonmbfmq22amc326am/node_modules/next/image.js
+      var next_image = __webpack_require__(4);
       var image_default = /*#__PURE__*/ __webpack_require__.n(next_image); // CONCATENATED MODULE: ./pages/nextjs.png
       /* harmony default export */ const nextjs = {
         src: "/_next/static/media/nextjs.cae0b805.png",
@@ -1030,12 +1030,8 @@
       /***/
     },
 
-    /***/ 5519: /***/ (
-      module,
-      __unused_webpack_exports,
-      __webpack_require__
-    ) => {
-      module.exports = __webpack_require__(7735);
+    /***/ 4: /***/ (module, __unused_webpack_exports, __webpack_require__) => {
+      module.exports = __webpack_require__(4563);
 
       /***/
     },
@@ -1045,7 +1041,7 @@
     /******/ var __webpack_exec__ = (moduleId) =>
       __webpack_require__((__webpack_require__.s = moduleId));
     /******/ __webpack_require__.O(0, [2888, 9774, 179], () =>
-      __webpack_exec__(3428)
+      __webpack_exec__(6324)
     );
     /******/ var __webpack_exports__ = __webpack_require__.O();
     /******/ _N_E = __webpack_exports__;

Diff for 5496-HASH.js

Diff too large to display

Diff for app-page-exp..ntime.dev.js

failed to diff

Diff for app-page-exp..time.prod.js

Diff too large to display

Diff for app-page-tur..time.prod.js

Diff too large to display

Diff for app-page-tur..time.prod.js

Diff too large to display

Diff for app-page.runtime.dev.js

failed to diff

Diff for app-page.runtime.prod.js

Diff too large to display

Diff for app-route-ex..ntime.dev.js

Diff too large to display

Diff for app-route-ex..time.prod.js

Diff too large to display

Diff for app-route-tu..time.prod.js

Diff too large to display

Diff for app-route-tu..time.prod.js

Diff too large to display

Diff for app-route.runtime.dev.js

Diff too large to display

Diff for app-route.ru..time.prod.js

Diff too large to display

Diff for server.runtime.prod.js

Diff too large to display

Commit: 4b383dc

…n paths See the commit files for more commentary

ijjk added the created-by: Turbopack team PRs by the Turbopack team. label Aug 21, 2024

arlyon marked this pull request as ready for review August 21, 2024 12:13

validate path lengths before writes and attempt to eagerly convert wi…

4b383dc

…n paths See the commit files for more commentary

arlyon force-pushed the arlyon/attempt-fix-too-long-file branch from 4167d4f to 4b383dc Compare August 21, 2024 14:07

This was referenced Aug 21, 2024

hash the font file name if it exceeds our max file size #69144

Open

refactor some of the font code #69146

Open

arlyon requested review from wbinnssmith and bgw August 23, 2024 17:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

validate path lengths before writes and attempt to eagerly convert win paths #69141

validate path lengths before writes and attempt to eagerly convert win paths #69141

arlyon commented Aug 21, 2024 •

edited

Loading

arlyon commented Aug 21, 2024 •

edited

Loading

ijjk commented Aug 21, 2024 •

edited

Loading

ijjk commented Aug 21, 2024 •

edited

Loading

validate path lengths before writes and attempt to eagerly convert win paths #69141

Are you sure you want to change the base?

validate path lengths before writes and attempt to eagerly convert win paths #69141

Conversation

arlyon commented Aug 21, 2024 • edited Loading

What?

Why?

How?

arlyon commented Aug 21, 2024 • edited Loading

ijjk commented Aug 21, 2024 • edited Loading

Failing test suites

ijjk commented Aug 21, 2024 • edited Loading

Stats from current PR

arlyon commented Aug 21, 2024 •

edited

Loading

arlyon commented Aug 21, 2024 •

edited

Loading

ijjk commented Aug 21, 2024 •

edited

Loading

ijjk commented Aug 21, 2024 •

edited

Loading