Form - Screaming Frog internal_html.csv upload

This form node is used to trigger the workflow.

It contains three input fields:

• Name of the website
• Short description of the website
• Screaming Frog export containing the internal URLs

It is recommended to use the internal_html.csv export, but internal_all.csv will also work, as the workflow includes a filter to process only indexable URLs.

Extract data from Screaming Frog file

This node extracts data from the CSV file provided by the user.

It produces an output that is easily usable in the following nodes.

⚠️ Caution:
If the uploaded file is not the expected Screaming Frog export, the workflow will still proceed but will likely fail in the next steps due to missing required fields.

Set Useful Fields

This node sets 7 key fields from the Screaming Frog export:

• url → from the "Address" column
• title → from the "Title 1" column
• description → from the "Meta Description 1" column
• status → from the "Status Code" column
• indexability → from the "Indexability" column
• content_type → from the "Content Type" column
• word_count → from the "Word Count" column

Multi-language compatibility
If you're using Screaming Frog in French, Italian, German, or Spanish, the column names will be different.
However, the workflow is designed to handle this, so it will still work correctly! 🥳

Filter URLs

This filter node is used to keep only the URLs that meet the following conditions:

• status = 200
• indexability = indexable
• content_type contains text/html

These filters are even more useful if the uploaded file is an internal_all.csv instead of an internal_html.csv.

Tips:

You can add more filters to refine the URLs included in your llms.txt file.

💡 Examples:

• Filter by word count → Ensure pages contain enough text content.
• Filter by URL path → Keep only specific folders or categories in the llms.txt file.
• Filter by meta description → Exclude URLs without a meta description, as this field will be used in the llms.txt file to describe each piece of content.

Text Classifier

🚫 This node is deactivated by default in the template.

You can enable it if you want to add a more "intelligent" 🤓 filter to refine the URLs included in the llms.txt file, helping LLMs discover and prioritize valuable content.

How It Works:

This node has two outputs:

• useful_content → Pages that are likely to contain high-quality content, making them suitable for inclusion in a file that aids content discovery for an LLM.
• other_content → Pages that should not be included (e.g., pagination or low-value content).

You can modify the description in the node to fine-tune the classification according to your needs.

Input Fields:

• url → {{ $json.url }}
• title → {{ $json.title }}
• description → {{ $json.description }}
• word_count → {{ $json.word_count }}

Why use an LLM?

A language model (LLM) can analyze the URL, title, and description to identify pages that most likely contain meaningful and relevant content.
This allows it to prioritize valuable pages and structure the data for better content discovery and training purposes.

For large websites

If you have a very large website, consider using a Loop Over Items node to make the workflow more robust and ensure all pages are processed.
Also, using a Loop Over Items node make it easier to handle:

• Timeouts
• API quotas
• Other scalability issues

Tokens usage

Finally, keep in mind that more pages mean more tokens and more billed LLM API calls.

Set Field - llms.txt Row

This node sets the row format for the llms.txt file.

Row Structure:

Each row follows this format:

• - [title](link): description

If the URL has no description (from the Meta Description in the Screaming Frog export), the row will be:

• - [title](link)

Summarize - Concatenate

This node concatenates all the output from the previous node, ensuring each row is on a separate line.

Set Fields - llms.txt Content

This node sets the content of the llms.txt file using:

• The website title provided in the form (first node).
• The website description provided in the form (first node).
• The output from the previous node, which includes all the URLs, their titles, and their descriptions that will appear in the llms.txt file.

Generate llms.txt file

This node creates the llms.txt file, which can be downloaded directly within n8n.

upload file anywhere

Instead of downloading the file directly from the n8n workflow, you can replace this node node with a Drive node (e.g., Google Drive or OneDrive) to upload the llms.txt file to a folder of your choice.

Name the file properly (e.g., include the website name) to make it easier to find and distinguish between files when working on multiple websites.

Generate AI-Ready llms.txt Files from Screaming Frog Website Crawls

This workflow helps you generate an llms.txt file (if you're unfamiliar with it, check out this article) using a Screaming Frog export.

Screaming Frog is a well-known website crawler.
You can easily crawl a website. Then, export the "internal_html" section in CSV format.

How It Works:

A form allows you to enter:

• The name of the website
• A short description
• The internal_html.csv file from your Screaming Frog export

Once the form is submitted, the workflow is triggered automatically, and you can download the llms.txt file directly from n8n.

Downloading the File

Since the last node in this workflow is "Convert to File", you will need to download the file directly from the n8n UI.
However, you can easily add a node (e.g., Google Drive, OneDrive) to automatically upload the file wherever you want.

AI-Powered Filtering (Optional):

This workflow includes a text classifier node, which is deactivated by default.

• You can activate it to apply a more intelligent filter to select URLs for the llms.txt file.
• Consider modifying the description in the classifier node to specify the type of URLs you want to include.

How to Use This Workflow

1. Crawl the website you want to generate an llms.txt file for using Screaming Frog.
2. Export the "internal_html" section in CSV format.
   
3. In n8n, click "Test Workflow", fill in the form, and upload the internal_html.csv file.
4. Once the workflow is complete, go to the "Export to File" node and download the output.

That's it! You now have your llms.txt file!

Recommended Usage:
Use this workflow directly in the n8n UI by clicking 'Test Workflow' and uploading the file in the form.

Generate AI-Ready llms.txt Files from Screaming Frog Website Crawls

Purpose and structure

This folder contains the Generate AI-Ready llms.txt Files from Screaming Frog Website Crawls workflow. Its executable nodes use convertToFile, extractFromFile, filter, formTrigger, lmChatOpenAi, noOp, set, summarize, textClassifier. The table and connection map below are derived from the JSON, not an inferred guarantee of behavior.

Setup

Import inactive. Configure your own credentials and replace REPLACE_WITH_* values. Review external destinations and Code nodes before running.

Entry points

Form - Screaming frog internal_html.csv upload

Verification

Sanitized source candidate; not runtime-tested. This overview is generated from node metadata. Read the folder README for the connection map, credential types, and license.

Attribution

https://github.com/Zie619/n8n-workflows/blob/94007c1445d9258a7da116646b79473e7c7c3282/workflows/Filter/1791_Filter_Summarize_Create_Triggered.json MIT; preserve source notices.