ShipKit Build Memory Optimization Guide

ShipKit Build Memory Optimization Guide

This guide addresses out-of-memory (OOM) issues during builds, particularly with large dependencies like @huggingface/transformers and other memory-intensive libraries.

🚨 Common Build Issues

Symptoms

  • Error: Command "pnpm run build" exited with SIGKILL
  • Out of Memory errors during build
  • Large WASM files (21.6 MB) won't be precached
  • Build timeouts on Vercel/CI platforms

Root Causes

  1. Large Dependencies: Libraries like @huggingface/transformers, three.js, remotion
  2. WASM Files: Heavy WebAssembly files (especially from AI/ML libraries)
  3. Insufficient Memory Allocation: Default Node.js memory limits
  4. Build Configuration: Unoptimized webpack/Next.js settings

📊 Current Dependency Analysis

Run the bundle analyzer to understand your current situation:

npm run analyze:bundle

Memory-Intensive Dependencies

  • @huggingface/transformers (21.6 MB WASM file)
  • three.js and @react-three/* (3D graphics)
  • remotion (video processing)
  • @opentelemetry/* (monitoring)
  • monaco-editor (code editor)
  • playwright (browser automation)

🔧 Optimization Solutions

1. Node.js Memory Configuration

Local Development

# Use memory-optimized build
npm run build:memory-optimized

# Or with custom memory settings
NODE_OPTIONS="--max-old-space-size=6144 --max-semi-space-size=1024" npm run build

Vercel/Production

# Use Vercel-optimized build
npm run build:vercel

# Environment variables (set in Vercel dashboard)
NODE_OPTIONS="--max-old-space-size=8192 --max-semi-space-size=512"

2. Next.js Configuration Optimizations

The next.config.ts has been optimized with:

experimental: {
  // Memory optimization settings
  workerThreads: false,
  esmExternals: true,
  cpus: 1, // Limit to single CPU core
  isrMemoryCacheSize: 0,
  largePageDataBytes: 128 * 1000, // 128KB limit
}

3. Webpack Bundle Splitting

Optimized chunk splitting prevents large bundles:

splitChunks: {
  maxSize: 200000, // 200KB max chunk size
  cacheGroups: {
    // Split transformers into smaller chunks
    transformers: {
      test: /[\\/]node_modules[\\/]@huggingface[\\/]transformers[\\/]/,
      name: "transformers",
      chunks: "async", // Only load when needed
      priority: 15,
      maxSize: 50000, // Very small chunks
    },
    // Other optimizations...
  }
}

4. PWA Configuration for Large Files

Updated PWA config to handle large WASM files:

// src/config/nextjs/with-pwa.ts
const pwaConfig = {
  maximumFileSizeToCacheInBytes: 25 * 1024 * 1024, // 25MB
  exclude: [/\.wasm$/, /\.bin$/, /onnx/, /transformers/],
  runtimeCaching: [
    {
      urlPattern: /\.wasm$/,
      handler: "CacheFirst",
      options: {
        cacheName: "wasm-cache",
        expiration: {
          maxEntries: 10,
          maxAgeSeconds: 60 * 60 * 24 * 30, // 30 days
        },
      },
    },
  ],
};

5. Dependency Externalization

Heavy libraries are externalized on the server:

// next.config.ts
config.externals = [
  {
    "@huggingface/transformers": "@huggingface/transformers",
    three: "three",
    "@react-three/fiber": "@react-three/fiber",
    remotion: "remotion",
    "better-sqlite3": "better-sqlite3",
  },
];

⚠️ Important: Do NOT externalize sharp as it's required at runtime by:

  • Payload CMS for image processing
  • Image optimization services
  • Server-side image transformations

🚀 Build Scripts

Available Scripts

{
  "build": "NODE_OPTIONS=\"--max-old-space-size=6144\" next build",
  "build:memory-optimized": "NODE_OPTIONS=\"--max-old-space-size=6144 --max-semi-space-size=1024\" next build",
  "build:vercel": "NODE_OPTIONS=\"--max-old-space-size=8192 --max-semi-space-size=512 --optimize-for-size\" next build",
  "analyze:bundle": "node scripts/analyze-bundle.js"
}

Usage

# Local development
npm run build:memory-optimized

# Production/Vercel
npm run build:vercel

# Analyze dependencies
npm run analyze:bundle

🔍 Monitoring and Troubleshooting

Memory Usage Monitoring

# Check memory before build
npm run prebuild

# Monitor Node.js memory
node --trace-gc --max-old-space-size=8192 node_modules/.bin/next build

# Analyze bundle size
npm run analyze:both

Debugging OOM Issues

  1. Check Current Memory Usage:

    npm run analyze:bundle

  2. Identify Large Dependencies:

    npx webpack-bundle-analyzer .next/static/chunks/*.js

  3. Monitor Build Process:

    # Enable verbose logging
DEBUG=* npm run build:memory-optimized

Common Solutions

| Issue | Solution | | -------------------- | ------------------------------------------------------ | | WASM files too large | Increase maximumFileSizeToCacheInBytes in PWA config | | Build OOM on Vercel | Use build:vercel script with higher memory limits | | Large bundle size | Enable chunk splitting and externalization | | Slow builds | Disable source maps in production | | Memory leaks | Use --optimize-for-size flag |

🎯 Best Practices

1. Dynamic Imports for Heavy Libraries

// Instead of static imports
import { AutoTokenizer } from "@huggingface/transformers";

// Use dynamic imports
const loadTransformers = async () => {
  const { AutoTokenizer } = await import("@huggingface/transformers");
  return AutoTokenizer;
};

2. Conditional Loading

// Only load AI features when needed
const AIComponent = lazy(() =>
  import("./ai-component").then((module) => ({
    default: module.AIComponent,
  })),
);

3. Environment-Based Optimization

// next.config.ts
const config = {
  experimental: {
    workerThreads: process.env.NODE_ENV === "development" ? true : false,
    cpus: process.env.NODE_ENV === "development" ? undefined : 1,
  },
};

4. Build Environment Configuration

Create .env.build with optimized settings:

NODE_OPTIONS="--max-old-space-size=8192 --max-semi-space-size=512 --optimize-for-size"
NEXT_TELEMETRY_DISABLED=1
SKIP_ENV_VALIDATION=1
NEXT_DISABLE_SOURCEMAPS=true

📋 Quick Fixes Checklist

  • [ ] Use build:vercel script for production builds
  • [ ] Increase Node.js memory limits (--max-old-space-size=8192)
  • [ ] Enable chunk splitting for large libraries
  • [ ] Externalize heavy dependencies on server
  • [ ] Configure PWA for large WASM files
  • [ ] Disable source maps in production
  • [ ] Use dynamic imports for AI/ML libraries
  • [ ] Monitor build with bundle analyzer
  • [ ] Test build locally before deployment

🆘 Emergency Fixes

If you're experiencing immediate build failures:

  1. Temporary workaround:

    # Disable problematic features
DISABLE_HUGGINGFACE_TRANSFORMERS=true npm run build

  2. Reduce memory usage:

    # Use single-threaded build
NODE_OPTIONS="--max-old-space-size=8192 --optimize-for-size" npm run build

  3. Skip non-essential features:

    # Skip linting and type checking
SKIP_LINT=true SKIP_TYPE_CHECK=true npm run build

📚 Additional Resources

🤝 Contributing

If you discover new optimization techniques or encounter issues, please:

  1. Test the solution locally
  2. Update this documentation
  3. Share findings with the team
  4. Monitor build performance metrics

Last Updated: 2024-01-XX Build Environment: Node.js 18+, Next.js 15+, Vercel