auto-update-llms-robots-txt

Auto-Update llms.txt and robots.txt

When a page file is created, edited, or deleted, you MUST update both llms.txt and robots.txt to reflect the current state of all crawlable pages.

Detection Rules

1. Identify Protected/Authenticated Routes

A page is considered protected (should be disallowed) if ANY of the following conditions are met:

By Path Pattern:

• Located under /auth/, /admin/, /dashboard/, /account/, /settings/, /profile/, /private/, /internal/, /api/
• Path contains [...auth], (auth), (protected), (private), (admin)
• Located under route groups like (dashboard), (admin), (authenticated)

By Code Analysis:

• Uses authentication HOCs: withAuth, withSession, requireAuth, ProtectedRoute, AuthGuard
• Uses auth hooks: useAuth, useSession, useUser, useProtectedRoute
• Has middleware protection (check for middleware.ts/js in parent directories)
• Contains auth checks: getServerSession, getSession, auth(), requireAuthentication
• Uses route guards or navigation guards (Vue Router, Angular)
• Has export const config = { auth: true } or similar metadata
• Contains redirects to login pages for unauthenticated users

By File/Folder Naming:

• Files prefixed with _ (often private in some frameworks)
• Located in folders named protected/, private/, secure/

2. Identify Public Routes

A page is considered public (should be allowed) if:

• No authentication logic detected
• Explicitly marked as public: export const config = { auth: false }
• Common public pages: /, /about, /contact, /pricing, /blog/*, /docs/*, /faq, /terms, /privacy, /help

File Locations

Look for existing files in these locations (in order of priority):

llms.txt:

1. /public/llms.txt
2. /static/llms.txt
3. /llms.txt (root)

robots.txt:

1. /public/robots.txt
2. /static/robots.txt
3. /robots.txt (root)

If files don't exist, create them in /public/ (or /static/ for frameworks that use it).