Technical SEO and JSON-LD markup

Technical SEO is setting up a site for the search robot: robots.txt, sitemap.xml, canonical addresses, noindex. JSON-LD markup says what a page is about.

First, get into the house

In the last lesson you saw that a search engine goes through three steps: crawling, indexing and ranking. This lesson is about the first two. It is technical but not difficult: it comes down to three small files, one meta tag and one block of markup. If you get these wrong, nobody will see the best texts in the world. A common “surprise” of a new site is that during development the robots were shut out so as not to show a raw version, and after launch nobody remembered to open the door.

robots.txt: what the robot may do and need not do

What it is. A text file that lives strictly at /robots.txt in the root of the site (https://riverbend.example/robots.txt). Honest search robots read it first and only then go around the site. The rules are written in groups: for which robot (User-agent), what is forbidden (Disallow) or allowed (Allow).

User-agent: *
Disallow: /account/
Sitemap: https://riverbend.example/sitemap.xml

This says: all robots must keep out of the customer account section, and the sitemap is at such-and-such an address. The asterisk means “all”. For particular robots you can write separate groups (User-agent: Googlebot, User-agent: Bingbot).

Where it comes from. The file was devised in 1994 by Martijn Koster for the early search engines, so that site owners could tell robots where not to go. For almost thirty years it was an informal agreement. Only in 2022 was it made a standard: RFC 9309, which is among the sources. Google and Bing support it, and each has its own additions (Yandex, for example, has Clean-param for addresses with parameters).

What robots.txt does not do. This matters, because it is where people go wrong most often:

• It doesn’t protect. It is a request, not a lock. Honest robots will obey, an attacker won’t. Close what is secret with a password and access rights on the server.
• It is public. Anyone can open /robots.txt. If you list secret sections in it, you tell everyone about them.
• It controls access, not display. If a page is blocked in robots.txt but other sites link to it, the engine may show a bare address with no description. To keep a page from being shown you need another tool: noindex.
• One mistake closes everything. The line Disallow: / forbids the entire site. Check the file after every edit.

sitemap.xml: a list for the robot

What it is. A file (most often /sitemap.xml) with a list of the addresses you want to show the search engine. The XML format is simple:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://riverbend.example/</loc><lastmod>2026-10-27</lastmod></url>
  <url><loc>https://riverbend.example/services/</loc><lastmod>2026-10-20</lastmod></url>
</urlset>

Only loc is required. lastmod (the date of the last change) is useful if you give it honestly. The other tags (priority, changefreq) Google ignores. One file holds up to 50,000 addresses and 50 megabytes.

What to understand. A sitemap is a hint, not a command: it helps the robot find pages (especially ones nothing links to) but doesn’t force indexing. Name it in robots.txt with a Sitemap: line and submit it in Search Console and Bing Webmaster Tools. Include only the pages you want: take out those closed by noindex or redirected.

The canonical page and duplicates

One page is often available at several addresses: http:// and https://, with www and without, with a slash at the end and without, with parameters (?utm_source=..., you know the tags from the lesson on social networks). For a search engine these are different pages with the same content: duplicates. They split the strength of links between them and confuse the engine.

What to do:

1. One main version. Choose: https, with www or without. Send the other versions to the main one with a 301 redirect (you studied status codes in the part on HTTP: 301 means “moved permanently”).
2. rel="canonical". In the head of each page write <link rel="canonical" href="https://riverbend.example/services/">, and make the address full. It is a hint: “the main version of this page is this one”. A page may point to itself.
3. A uniform look of addresses. Decide on the slash at the end and on lowercase letters, and stick to it.

noindex: how to keep a page out of search

Not every page of a site is needed in search: the “Thank you” after a request, the customer account, internal search results, the cart. To keep the engine from storing them, put a meta tag in the head:

<meta name="robots" content="noindex">

Or the same in the server’s response header (X-Robots-Tag: noindex) if the file isn’t HTML, a PDF for example.

The main trap, which the diagram below shows: for noindex the robot has to open the page. If you blocked it in robots.txt, the robot won’t open it and won’t see the tag. So the rule is: pages that must not be in search should stay open to the robot and carry noindex. Keep robots.txt for sections the robot just has no need to visit (huge service directories), and don’t count on it for protection.

And for a customer account you need a password sign-in: that is protection, while noindex is only a request not to show.

The spider game

Open the diagram. It is an invented site of seven pages and the links between them. The robot begins at the home page and follows links. Block pages in robots.txt, set noindex, switch on a sitemap, and see what the robot finds, reads and remembers. The goal: the pages you need are reachable and in the index, the service ones stay out of the index, and no rule gets in another’s way.

there is a sitemap.xml with all the addresses the home page links to the old offer

• Home /

  Disallow in robots.txt noindex

  In the index. Crawled, and may enter the index.

• Services /services/

  Disallow in robots.txt noindex

  In the index. Crawled, and may enter the index.

• Prices and booking /prices/

  Disallow in robots.txt noindex

  In the index. Crawled, and may enter the index.

• An article on boundaries /blog/boundary/

  Disallow in robots.txt noindex

  In the index. Crawled, and may enter the index.

• The customer account /account/ (a service page: not needed in search)

  Disallow in robots.txt noindex

  In the index. Crawled, and may enter the index.

• The “Thank you” page /thanks/ (a service page: not needed in search)

  Disallow in robots.txt noindex

  In the index. Crawled, and may enter the index.

• The old offer (no links to it) /old-offer/

  Disallow in robots.txt noindex

  Not in the index. The robot did not find this page: no link leads to it and it is not in the sitemap.

What is wrong (3): /account/: a service page will enter the index; /thanks/: a service page will enter the index; /old-offer/: a page you need was not found.

The site is invented. The rules are those the search engines state: robots.txt controls access, not display; to keep a page from being shown, leave it open and set noindex.

Check yourself: what is the right way to keep the “Thank you” page out of search?

Leave it open to the robot and put a noindex meta tag in the head. If you also block it in robots.txt, the robot won’t see the tag, and the address may still appear in results.

Languages: hreflang

For a site in several languages (the “languages” field of the brief), the head of each version names its “relatives” with <link rel="alternate" hreflang="ru" href="..."> tags (you studied this in the part on HTML and page languages). There are three rules: each version links to all versions, itself included; addresses are full; and pages link to each other both ways. Then Google, Bing and Yandex show a person the page in their language instead of at random.

Markup: telling the search engine what a page is

A search engine reads the text of a page and guesses what is what. The guesses aren’t always right: is “2026” a year or a price? Markup removes the question.

schema.org is a shared vocabulary of types and properties: Organization, Person, Event, Product, Article and hundreds of others. It was created in 2011 by the biggest search engines (Google, Microsoft, Yahoo, joined by Yandex). JSON-LD is a way to write that data: as ordinary JSON inside a script tag of type application/ld+json. The JSON-LD standard was adopted by the W3C. Google recommends exactly this format, and Bing and Yandex also understand schema.org.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Project name",
  "url": "https://riverbend.example/",
  "email": "hello@riverbend.example",
  "telephone": "+1 000 000 0000"
}
</script>

What it gives: the engine is clearer about what the page is and who stands behind it; and for some types your page may get rich display in the results (a rating, a price, an event date). What it doesn’t give: guarantees. Google decides itself whether to show a rich result and changes over time which kinds it supports: see the current list in its documentation.

Rules of honest markup.

1. Mark up what is visible on the page. Opening hours that aren’t on the page can’t be given. False markup breaks Google’s rules and can lead to penalties.
2. Choose the type by what it is. An expert is Person or ProfessionalService, a company Organization, a place LocalBusiness (the next lessons), a shop Store with Product items, a festival Event, a service SoftwareApplication.
3. Check it. Google’s Rich Results Test and the Schema Markup Validator can be found by those names; they show syntax errors.
4. Don’t multiply. One main entity per page, with the rest linked to it.

Practice

Prepare the technical parts of the site.

1. robots.txt: a User-agent line, only service sections blocked, a Sitemap line with a full address.
2. Addresses and sitemap.xml: a full address for each section from the brief, and the file itself.
3. Pages kept out of the index: at least two, at least one through noindex.
4. JSON-LD in the head: @context, @type, the project’s name, address and a contact from the brief.

The check looks at the form of the files and the markup block. Whether the type suits your project is for you to decide: open schema.org and find a type that describes you more precisely than Organization.