LLMs.txt Generator

VIDEO WALKTHROUGH

LLMs.txt Generator: setup guide

2:42

Recorded on version 1.0.9 · Narrated walkthrough · English captions

Jump to a chapter 8 chapters

1. 0:00Set up LLMs.txt Generator
2. 0:14Open settings and choose scope
3. 0:35Describe the public site
4. 0:53Check storefront URLs
5. 1:13Choose what to include
6. 1:35Decide on the JSON endpoint
7. 1:57Save and verify your setup
8. 2:18Your setup checklist

Read the full video transcript

LLMs.txt Generator 1.0.9 — Magento Admin setup

00:00:00 — Set up LLMs.txt Generator
Welcome to the Agentic Ecom L L M S dot text Generator Admin guide. We'll walk through the main setup screens and the checks to make before using the module. This guide assumes the extension is already installed.

00:00:14 — Open settings and choose scope
Start in Magento Admin and open Stores, Configuration. Choose LLMs.txt Generator. Check the Scope selector before editing. Use the appropriate website or store view where the setting supports it. The screen values here are a demonstration; enter the settings and service details that belong to your own store.

00:00:35 — Describe the public site
In General Settings, write a factual Site Description and review the short and full file URL keys. These files describe your public catalogue to tools that choose to read them. Publishing the files does not guarantee that an assistant will fetch, use or cite your store.

00:00:53 — Check storefront URLs
For Luma, use the standard storefront configuration. If you use Headless Storefront Mode, set the correct public frontend domain and verify the generated product links. The public file must point readers to the customer-facing storefront, rather than an internal API or an unrelated demonstration domain.

00:01:13 — Choose what to include
Open Content to Include. Decide whether products, categories, C M S pages and blog posts belong in the output. Review Deep Attribute Export carefully: include only product information that is suitable for public access. Check the generated file after changing catalogue content or inclusion settings.

00:01:35 — Decide on the JSON endpoint
The JSON Structured Endpoint is optional. Enable it only if you want that structured public output and have reviewed what it exposes. After saving, check the short summary, the cached full file and any enabled JSON endpoint. Verify the content and links in the intended store view before pointing an integration at them.

00:01:57 — Save and verify your setup
When your required settings are complete, use the screen's Save control and wait for Magento's confirmation. Then reopen the record or configuration and check the saved values at the intended scope. If the storefront still shows earlier settings, refresh the relevant caches. Test your own configuration before using it with customers.

00:02:18 — Your setup checklist
That's the Admin setup walkthrough for L L M S dot text Generator. Check permissions and scope, review the module-specific settings, then save and verify a small staging example. Use the written guide at user guides dot Agentic Ecom dot net for installation details, complete field references and any separate storefront integration steps.

Download videoEnglish captionsDownload transcript

robots.txt tells search crawlers what they may fetch. llms.txt is a proposed Markdown format for describing a site to tools that choose to read it. This module serves a short summary from catalogue data, a fuller file cached per store view, and an optional product JSON feed. Publishing these files does not ensure that an assistant will fetch or cite them.

Install

Install LLMs.txt Generator with Composer using your AgenticEcom licence key (find it on your My Licences page — one key unlocks every module you own):

# 1. Register the AgenticEcom repository (once per store)
composer config repositories.agenticecom composer https://repo.agenticecom.net
composer config --global --auth http-basic.repo.agenticecom.net <YOUR-LICENCE-KEY> x

# 2. Require this module
composer require agenticecom/module-llmstxt

# 3. Apply it
php bin/magento setup:upgrade && php bin/magento setup:di:compile && php bin/magento cache:flush

Have the whole platform?

Install every module in one command with composer require agenticecom/suite — see Install the complete suite.

Compatibility

Magento

Check the exact release against your Magento Open Source version.

PHP

Check the exact package requirements and test your PHP version.

Per store view

Independent URL keys and content per store/locale.

Refresh

The short file is generated on request; the full file is cached and refreshed by cron every six hours.

The three files it serves

/llms.txt

The short manifest — your site description plus a pointer to the full file. This is the entry point a crawler fetches first.

# LLM Crawler Information

User-Agent: *
Disallow:

## Application Description
AgenticEcom — a Magento 2.4.9 store offering 40+ AI-native extensions…

## Detailed Information
Map: https://your-store.com/llms-full.txt

/llms-full.txt

The full content as Markdown — a linked list of every product, category, CMS page and blog post (each section is individually toggleable). In Deep mode each product also gets its specifications, variants, live stock and images.

# Full Site Content

## Products
- [Starter Bundle](https://your-store.com/starter-bundle) - Eight essential modules…

## Categories
- [Bundles & Packages](https://your-store.com/bundles)

/llms-products.json

An opt-in structured export — every visible product with deep attributes, configurable variants, stock and images, as JSON formatted for vector-DB ingestion or direct context injection into an AI chat. Disabled by default; returns 403 until you switch it on.

{ "products": [ { "id": 2041, "sku": "AE-BUN-STARTER",
  "name": "Starter Bundle", "price": 795, "stock": {"in_stock": true},
  "attributes": { … }, "variants": [ … ] } ] }

Refresh timing

/llms.txt is generated when requested. /llms-full.txt is expensive to build, so the module caches it per store view with a six-hour lifetime and schedules a cache refresh every six hours. A cold cache may be generated during a request. /llms-products.json, when enabled, is generated from source data but sends a public one-hour cache header. Check the output after catalogue edits if immediate publication matters; a successful Magento save does not itself prove that every cached or downstream copy has refreshed.

Set it up

1. Go to Stores → Configuration → AgenticEcom · SEO & Content → LLMs.txt Generator.

2. Under General Settings, switch Enable LLMs.txt to Yes and write a one-paragraph Site Description — this is what an assistant reads first, so describe what you sell and who for.

3. Under Content to Include, choose which sections appear in the full file: Products, Categories, CMS Pages, Blog Posts. Turn on Enable Deep Attribute Export if you want full specs, variants, stock and images per product.

4. (Optional) Under Agentic Chat Integration, enable the JSON Structured Endpoint to expose /llms-products.json for vector-DB or AI-chat ingestion.

5. Visit /llms.txt on your storefront to confirm it renders.

Rename the URLs safely

The llms.txt URL Key and llms-full.txt URL Key fields let you change the storefront paths (e.g. a German store at /llms-ai.txt), per store view. When you rename one, a 301 redirect from the old slug is written automatically, so any crawler that already indexed the old path keeps working.

Only public content — by design

Products

Filtered to enabled + visible-in-site products only. A disabled or “not visible individually” product never appears in any of the three files.

Categories, CMS & blog

Categories filtered to active, CMS pages to active, blog posts to published — so nothing hidden from the storefront leaks to a crawler.

No system attributes

Deep mode lists only attributes flagged visible on front (or used in listings), and skips a long denylist — meta_*, layout/design fields, tax class, cost — so internal data stays internal.

JSON is opt-in

The structured /llms-products.json endpoint is off by default and returns 403 until you explicitly enable it, so you decide when full machine-readable product data goes public.

FAQ

What actually is llms.txt — do I need it?

It’s a proposed convention (think robots.txt, but for AI assistants) that gives LLM-powered tools a clean, authoritative description of your site instead of leaving them to scrape rendered HTML. It gives a tool that fetches the file a structured source for your store description and selected content. Whether a particular assistant uses or cites it depends on that assistant.

Will it expose disabled products, draft pages or internal fields?

No. Every section filters to active/visible content (enabled + visible-in-site products, active categories, active CMS and published blog posts), and Deep mode lists only front-visible attributes while skipping meta/layout/tax/cost fields. Hidden content stays hidden.

Is the JSON product feed public to everyone?

Only if you turn it on. /llms-products.json is disabled by default and returns 403 Forbidden until you enable JSON Structured Endpoint in the configuration — so exposing full machine-readable product data is always a deliberate choice.

Can I rename /llms.txt without breaking crawlers that already found it?

Yes — change the URL key in config and a 301 redirect from the previous path is written automatically, per store view, so existing references keep resolving.

Does it work with multiple store views / languages?

Yes — the description, content toggles and URL keys are all per store view, so each locale serves its own llms.txt from its own catalogue scope. Check exact-release compatibility before deployment.