security.txt, robots.txt and well-known endpoints

1 Introduction

Every website publishes a few small text files that are not pages at all. They are read by machines: browsers, mobile operating systems, search engines, and the people who look for security problems. Three of them show up in the Web & TLS section of a domain scan: security.txt, the /.well-known/ endpoints and robots.txt.

They do very different jobs. security.txt tells a researcher who found a vulnerability on your site where to report it. The well-known endpoints tie your domain to your mobile apps and your login provider. robots.txt asks crawlers to stay away from some paths, and often tells everyone else exactly where your admin pages are. None of them is hard to get right, and all three are public by design.

2022

security.txt becomes a standard (RFC 9116)

Required fields in security.txt: Contact and Expires

Protection offered by robots.txt: it is a request, not an access control

2 How It Works

All three files live at fixed addresses, so anyone (or any tool) can find them with a single request, without knowing anything about your site. The scan simply fetches them over HTTPS, as a visitor would, and reads what they say.

security.txt

At /.well-known/security.txt: who to contact about a vulnerability, and until when that information is valid.

/.well-known/

A reserved folder (RFC 8615) where standards put files that other software expects to find: app links, login discovery, password change.

robots.txt

At the site root: polite instructions for crawlers. Well-behaved bots follow them, everyone else can read them.

Public on purpose

Nothing here is a leak in itself: these files exist to be read. The questions a scan asks are different. Is the security contact present and still valid? Do the well-known files reveal more than they need to (internal app builds, test identities)? Does robots.txt list paths you would rather nobody looked at? Fixing these is mostly hygiene, but a missing or expired security.txt can mean a vulnerability report never reaches you.

3 Detailed Analysis

security.txt: your vulnerability disclosure contact

When someone finds a flaw on your site, they need a way to tell you. Without a published contact, reports go to a generic inbox, to social media, or nowhere. RFC 9116 fixes this with a plain text file served over HTTPS. A minimal valid file:

Contact: mailto:[email protected]
Expires: 2027-06-30T23:00:00.000Z
Policy: https://example.com/security-policy
Preferred-Languages: en, fr
Canonical: https://example.com/.well-known/security.txt

Contact (required)

One or more lines, each a URI: mailto:, https: (a form or a bug bounty page) or tel:. Use a shared mailbox that someone actually reads, not a person who may leave. The scan flags a file with no Contact (medium) and a Contact that is not one of these URI types (low).

Expires (required)

A date in RFC 3339 format. Past that date, a researcher must treat the file as stale, so an expired file is about as useful as none: the scan rates it medium. A missing or unreadable Expires is low. The RFC recommends a date less than a year ahead, and the scan notes (info) a date further away. Put a renewal reminder in your calendar.

Policy and Encryption

Policy: links to your disclosure policy: what researchers may test, how fast you answer, whether you promise not to take legal action. Encryption: links to a public key (OpenPGP) so reports can be sent encrypted. Both are optional and both make good reports more likely.

Canonical and signing

Canonical: states the URL where the file is supposed to live. If it points somewhere other than where the file is served, the scan flags it (low), since the file may have been copied from another site. The file can also carry an OpenPGP cleartext signature, which lets a reader check it was not tampered with. The scan shows "PGP signed" when it sees one. Signing is recommended, not required.

Location

The standard place is /.well-known/security.txt. The scan also looks at the older /security.txt and notes (info) when the file only exists there. No file at all is reported as low.

Well-known endpoints: what your domain declares

The scan probes five standard addresses and keeps only real answers (a page that returns your site's HTML for any path is not counted as a JSON file):

/.well-known/assetlinks.json

Android App Links: which Android packages may open your links. The report lists the package names.

/.well-known/apple-app-site-association

The Apple equivalent (universal links). The report lists bundle IDs and Apple Team IDs.

/.well-known/openid-configuration

OpenID Connect discovery: your identity provider, its issuer and its login, token and key endpoints.

/.well-known/nodeinfo and /.well-known/change-password

nodeinfo describes fediverse software (Mastodon and similar). change-password should redirect to your password change page, so password managers can send users straight there.

Findings here are info only (mobile app identities declared, OpenID issuer advertised): they cost nothing in the score. What matters is the content. These files are often generated once and forgotten, and they can name internal or development builds (com.example.android.staff, ...debug) or old apps. List only the apps you ship publicly, and remove entries for retired ones. An advertised identity provider is normal, but it tells an attacker where your login lives, so make sure it enforces MFA and rate limiting.

robots.txt: not a security control

robots.txt (standardised as RFC 9309 in 2022) asks crawlers not to fetch some paths. Search engines respect it. Attackers read it as a map:

User-agent: *
Disallow: /admin/
Disallow: /staging-2024/
Disallow: /backup/
Disallow: /api/internal/

A Disallow line is a signpost

Every path above is now public knowledge. If /admin/ is only protected by "nobody knows the URL", it is not protected. Real protection is authentication, an IP allow-list or a VPN, and the path does not need to be in robots.txt at all.

Disallow does not hide a page from search

A disallowed URL can still appear in search results if other sites link to it. To keep a public page out of the index, use a noindex meta tag or header instead.

What the scan does with it

The raw file is folded under Web & TLS, and its rules for AI crawlers feed the AI crawling policy block. The scan does not judge your Disallow lines: open the file and read them yourself, it takes a minute.

A good robots.txt is short: it keeps crawlers out of search result pages, carts and other low-value URLs, and points to your sitemap. Anything sensitive is protected by the server, and left out of the file.

Check your domain on mlab.sh

A domain scan reads your security.txt against RFC 9116, probes the standard well-known endpoints and shows your robots.txt, in the Web & TLS section of the report.

Scan your domain on mlab.sh

4 Red Flags

An expired security.txt

Researchers are told to ignore it, so it tells them nothing. Update the Expires date (less than a year ahead) and set a reminder to do it again.

A security.txt with no Contact, or a contact nobody reads

The file exists only to give a contact. Send a test message to the address it lists and check that someone answers.

Disallow lines naming admin, staging or backup paths

You are publishing a list of places to look. Protect those paths with authentication, then remove them from robots.txt.

Internal or debug app builds in assetlinks or apple-app-site-association

Package names like .staff, .internal or .debug reveal apps that were never meant to be public. Keep only shipped apps in these files.

A Canonical URL pointing to another site

Usually a file copied from a template or a parent company. Researchers may report to the wrong team. Write your own file for each domain.

Related Modules

Your domain and AI: llms.txt, crawlers, MCP

What your domain tells AI systems and what it exposes to them: llms.txt, AI crawler rules, agent discovery and MCP servers.

Known vulnerabilities on your website

How a scan matches software versions to CVEs, what CVSS, EPSS and CISA KEV mean, and how to prioritise patching.

How to read a domain scan report

Score, grade, severities and report sections: how to read an mlab.sh domain scan and decide what to fix first.