SEO: crawling, indexing, ranking

SEO is work so that a search engine finds, understands and shows pages to those they help. It works in three steps: crawling, indexing, ranking.

How people find you when they don’t know you yet

In the earlier lessons you drew people in yourself, with emails and posts. But a huge number of those who need you don’t know you and expect nothing. They type a question into a search box. SEO (search engine optimization) is work so that a search engine finds, understands and shows your pages to those they help. It isn’t magic or hacking: it is a careful explanation to the engine of what your page is about, and honest work to make the page truly useful.

Where it comes from. The first search engines (Archie, AltaVista) appeared in the 1990s. In 1998 the Stanford students Sergey Brin and Larry Page described the PageRank algorithm in a paper and founded Google: their idea was that a page is important if important pages link to it. SEO as a profession grew from site owners quickly realizing that the flow of visitors depends on the place in the results. That gave rise both to honest work and to the deceit discussed below. Bing, the second large engine in English-speaking countries, also supplies results to other search products, DuckDuckGo and Yahoo among them. In Russia the search engine Yandex has been the main alternative since 1997.

The search engine’s three steps

A search engine works in three stages. It is very useful to keep them apart, because each has its own causes of failure.

1. Crawling. A program called a robot travels the web following links and downloads pages. It can find a page by a link from another page, or by a list of addresses you give it yourself (a sitemap). Google’s robot is called Googlebot, Bing’s is Bingbot. If the robot is kept out or the server answers with an error (you studied HTTP status codes in the part on the internet), the page goes no further.
2. Indexing. The engine analyzes the downloaded page: the text, headings, images, links, language, what is for what. Then it decides whether to record it in the index, the huge database from which it answers queries. It may decline: the page is empty, duplicates another, is closed by a noindex directive, or is judged low in quality.
3. Ranking. When a person types a query, the engine picks suitable pages from the index and orders them. How exactly, nobody outside knows: there are hundreds of factors, their weights are secret and change. The direction is known: how well the page answers the query, how useful and reliable it is, how fast and easy it is to open, and whether others link to it.

And a fourth, often forgotten: display. In the results the person sees a headline and a short description. They decide whether the person clicks. Here you have direct influence through the title and description (you met them in the lesson on the head and meta tags).

Check yourself: the page is on your site, but the search engine doesn’t show it. At which three stages could this have happened?

At crawling (the robot didn’t find it or couldn’t download it), at indexing (it downloaded but didn’t record it in the index) and at ranking (it is in the index but stands too low for the queries). The first two can be checked in Search Console or Bing Webmaster Tools.

What the person really wants

A search engine tries to give the person what they need. So the main thing to understand about a query is its intent: what the person really wants. Four typical intents: to find out (how to find the boundary of my lot), to choose (how a boundary survey differs from a topographic survey), to do (order a boundary survey, learn the price) and to find a specific place (a land surveyor, a phone number, an address). You have met them in another form: they are the See, Think and Do stages from the lesson on the funnel, and the pages for them differ: an article, a comparison, a service page, contacts.

Label a few queries in the diagram. Notice that one of them is honestly ambiguous: a query doesn’t always have a single intent.

1. “how to find my property boundaries”

   KnowChooseDoFind a place

2. “boundary survey vs topographic survey”

   KnowChooseDoFind a place

3. “boundary survey cost”

   KnowChooseDoFind a place

4. “land surveyor Riverbend phone number”

   KnowChooseDoFind a place

5. “what is a boundary expert witness”

   KnowChooseDoFind a place

6. “book a land surveyor site visit”

   KnowChooseDoFind a place

7. “what to do if my neighbor moved the fence”

   KnowChooseDoFind a place

8. “land surveyor services reviews”

   KnowChooseDoFind a place

Labeled: 0 of 8, agreeing with the author’s labels: 0. The author’s labels are an opinion too: check real queries in the statistics of Search Console and Bing Webmaster Tools.

The practical conclusion: one page is written for one intent. Don’t try to answer both “what is a boundary survey” and “boundary survey cost” on one page: the person with the first query needs an article, the one with the second needs a price and a form. For each page write down the query it is written for, and write for the person who typed it.

What depends on you at each step

• Crawling. You don’t shut out of the robot what is needed, you link pages to each other, you provide a sitemap, you make sure the server answers without errors.
• Indexing. Each page answers its own question and doesn’t duplicate another, service pages are kept out of the index, and the text makes sense.
• Ranking. You answer the question better than others, you show the author and the evidence, you make the site fast and comfortable on a phone.
• Display. You write a title and description that honestly describe the page and that people want to click.

The first two steps you’ll set up in the next lesson (robots.txt, the sitemap, canonical addresses, noindex). For now let’s deal with what the person and the search engine see on the page itself.

Title, description, headings

• Title (in the head). The page’s headline in the results; about 60 characters show. It holds the page’s topic and the project’s name, not “Home”: the word “home” tells neither a person nor a robot anything. Each page has its own.
• Description (a meta tag in the head). The text under the headline; about 155 to 160 characters show. It isn’t a “magic” factor: the engine may show its own text instead of yours. But a good description raises the share of clicks. Write so that it’s clear what is on the page and why to click.
• The top-level heading (h1). One per page, matching the title in meaning. Second and third level headings divide the text into parts (you studied the structure in the lesson on document headings).
• The address (URL). Short, in Latin letters, clear: /services/boundary-survey/, not /p?id=482.

Google, Bing and Yandex: what they share and how they differ

They share more than divides them: all of them crawl, index and rank, and the basics are the same. Differences that matter to a site:

• Tools. Google Search Console, Bing Webmaster Tools and, for Russia, Yandex Webmaster. In each you verify ownership of the site, submit a sitemap, see which pages are indexed and receive messages about errors. Connect the ones your audience uses, because you need to see each side. Bing Webmaster Tools can import a site from Search Console, so the second connection is quick.
• Region. Yandex ties the results more strongly to the region of the person and of the site, which matters for a local business in Russia. Google and Bing also have local search, built differently (more in the lesson on local SEO).
• Directives. In robots.txt Yandex has directives Google doesn’t, for example Clean-param (for addresses with parameters). Most of the main directives are common, and a common standard describes them (RFC 9309).
• Mobile and speed. Google has long indexed the mobile version of a page first. Bing and Yandex also take into account how a page looks and works on a phone.
• Answers from AI. All of them show generated answers above the ordinary results. That has a lesson of its own.

What no one knows except the search engines themselves: how much each factor weighs. Anyone who claims to know the exact formula is deceiving or mistaken. So a reliable strategy isn’t guessing the factors but working on what the engines themselves publicly advise: useful content, a clear site, honest methods.

Content and links

Content. In its guidance Google describes “helpful, reliable, people-first content”, written for people, not for search engines. In practice that means:

• It answers the question and does so first (the inverted pyramid from the lesson on text structure).
• It is written by someone who knows the subject, and it shows: the author is named, with qualifications and evidence. Google calls this E-E-A-T (experience, expertise, authoritativeness, trust), and topics on which a person’s health, money and rights depend “YMYL” (Your Money or Your Life): the bar for them is higher. Property law is exactly such a topic.
• It doesn’t repeat what has been said a hundred times but gives something of its own: your experience, examples, your own numbers.
• It isn’t stuffed with keywords. A text written “for the robot” (“cheap boundary survey boundary survey cost boundary survey Riverbend”) a person won’t read, and search engines have long recognized it.

Links. The idea of PageRank is that a link to your page from another site is a kind of recommendation. Good links (people link to you because you are useful) still help. But:

• Bought links break the rules of Google and Bing and can lead to demotion. Don’t buy them.
• Internal links (between your own pages) are entirely in your power: they help the robot find pages and the person go on. In a link write words that say where it leads, not “click here”.
• An honest link is one placed because someone found something useful at your site: a mention in a professional community, in an article, in a directory.

False promises. You’ll be offered “guaranteed first place”, “promotion in a week”, “thousands of links”. Remember that the search engine determines the result, not the contractor. Google itself says no one can guarantee a number-one ranking, and advises you to look for someone else if an SEO guarantees first place. An honest contractor promises work (a technical audit, fixes, texts) and measurable steps, never a position.

Practice

Prepare the page and the passports.

1. The head of the home page. A title of up to 60 characters with the project’s name, a description of 50 to 160 characters, a canonical address and (if the brief includes Russian) hreflang. One h1.
2. Page passports. For each section from the brief: an address, a title and a description. The title has the section’s name.
3. The connection. How you’ll verify ownership in Google Search Console and Bing Webmaster Tools, and which query the home page is written for.

The check looks at lengths, the project’s name, addresses and that all sections are present. Whether the descriptions are good in meaning is for you to decide: read each title and description as if you had seen it in the results. Would you click?