Documentation · Chapter 17

Generate discovery files without confusing security.

A sitemap lists preferred discoverable URLs. robots.txt guides compliant crawlers but never protects private information.

Every lesson starts from the problem, explains the mechanism, then gives a concrete verification.

LESSON OBJECTIVE

What you will be able to do

Generate sitemap and robots directives from real public URLs without mistaking them for authorization.

Generate from real static routes

seo:generate reads eligible GET routes and writes sitemap.xml and robots.txt. Dynamic routes need explicit URLs or a data-backed generator.

phpaml — zsh
aml seo:disallow /admin
aml seo:allow /admin/public
aml seo:generate

Exclude intentionally

Remove private, duplicate, temporary, and low-value routes from discovery. Authentication remains the protection for private content.

Submit and monitor

Submit the sitemap in each verified property, inspect priority URLs, and watch discovery, indexing, exclusions, and errors over time.

CHECKPOINT

Before you continue

Could a supposedly private URL still be opened directly despite its robots.txt rule? If yes, add real access control.