Asynchrony, fetch and JSON

fetch is a JavaScript function that sends an HTTP request and returns a promise of the response. Data usually arrives as JSON, a text format for objects.

Data that doesn’t arrive at once

So far the script has worked with what is already on the page. But a real site often takes data from a server: a list of events, free appointment times, the status of an order. And it sends data there: an inquiry, a review, a search. This always takes time: the request travels over the network, the server thinks, the answer travels back. A second, three, sometimes it never arrives.

If the script just stood still while it waited, the page would freeze: no scrolling, no clicks. To prevent that, everything connected with waiting is made asynchronous in JavaScript. JavaScript runs in a single thread, which means it does one thing at a time. But it hands the waiting for an answer to the browser (“send it when it comes”), gets on with everything else, and when the answer is ready the browser puts its handling in a queue, and the script reaches it in its turn. This queue is called the event loop, and it is why the handlers from the last lesson are called “sometime”, not at once.

Promises and async/await

A result that doesn’t exist yet is represented in JavaScript by a value called a promise. It is a box into which the result will be put later. A promise has three states:

• pending: there is no answer yet;
• fulfilled: a result has arrived;
• rejected: something failed.

Such things used to be done with “callbacks”: you passed a function and asked for it to be called when the result was ready. If one wait was followed by another, and then a third, you got a staircase of nested functions that was called callback hell. Promises arrived in 2015, and in 2017 a new notation, async/await, which lets you write waiting as ordinary code from top to bottom:

async function loadItems() {
  const response = await fetch('/items.json');   // wait for the answer, but the page is alive
  const items = await response.json();           // wait until the body has been read
  return items;
}

The word await goes before a promise and means “wait for the result, then carry on”. You can write it only inside a function declared with async. Such a function returns a promise itself, so it can be waited for somewhere else.

See what happens to the page while a request is on its way. Press the counter while you wait for the answer: in the ordinary mode the page lives, and in the “blocking” mode you’ll see how a page would look if it waited for the answer without asynchrony.

How the request ends success, code 200the server answered 404the connection dropped blocking mode (as if there were no asynchrony)

Send the request: the answer takes 3 seconds Start over

The page (main thread)

Press me: 0

The script can get on with its own business.

Network and server

silence

Promise

not created yet

Choose an outcome and send the request.

Check yourself: why doesn’t the page freeze while fetch waits for the answer?

The browser does the waiting, not the script. The script gets a promise and carries on, and when the answer is ready its handling joins the event queue.

fetch

fetch(address) sends an HTTP request (the very one you studied in the lesson on HTTP) and returns a promise of the response. The default method is GET. To send data, you add options:

const response = await fetch('/inquiry', {
  method: 'POST',
  body: new FormData(form)    // the form's data, as the browser would have sent it
});

The response (response) is like what the Network tab showed you: it has a status (the code), ok (a handy “the code is from 200 to 299”) and methods for reading the body: text(), json() and others. Reading the body is asynchronous too, which is why await stands before response.json() as well.

Here is a trap that nearly everyone falls into. fetch rejects its promise only on a network failure: no connection, address not found, connection cut. If the server answered, even with 404 or 500, that is an answer too, and the promise is fulfilled. So after every fetch you write a check:

if (!response.ok) {
  throw new Error('The server answered with the code ' + response.status);
}

Without it, the page will try to read an error page as data and show the visitor emptiness or strange text.

There is a security limit too: a browser lets a page request data from its own site. A request to another site goes through only if that site has allowed it in its headers (this is called CORS). If you see a red “blocked by CORS policy” in the console, the problem is on the server’s side, not in your code.

JSON

A server and a page exchange data as text, and the most common format for that text is called JSON (JavaScript Object Notation). Douglas Crockford popularized it in the early 2000s as a replacement for heavy XML, and it proved so simple that it became a standard (RFC 8259). It looks almost like the JavaScript objects and arrays you wrote in the first lesson:

[
  { "title": "Portrait session", "text": "An hour in the studio, twenty finished frames.", "price": "from £000" },
  { "title": "Reportage", "text": "Events, without getting in the way.", "price": "" }
]

But the rules are stricter:

• names and strings only in double quotes;
• objects, arrays, strings, numbers, true, false and null are allowed;
• no comments and no comma after the last item.

Many scripts fall over on that last comma: JavaScript forgives it, and JSON does not. Two commands turn text and values into each other: JSON.parse(text) reads JSON into an ordinary value, and JSON.stringify(value) writes a value as JSON. The response.json() method does JSON.parse on the body of the answer for you.

When something goes wrong

Three different troubles look different:

1. There is no network. fetch is rejected (the console says TypeError: Failed to fetch), and catch runs.
2. The server answered with an error. The promise is fulfilled and response.ok is false: the check above turns this into an error just like the first.
3. The answer is wrong. It isn’t JSON, or it lacks the field you need. response.json() rejects by itself, and missing fields you have to check.

Everything is wrapped in try ... catch ... finally. The try holds what may break. The catch receives the error. And the finally block runs in every case; there you remove the loading indicator, so it doesn’t hang forever after a failure.

What the visitor sees matters more than what the console sees. They need three states and honest text:

• loading: “Loading the fresh list…” (a region with role="status" lets screen readers announce it to blind visitors);
• success: the data;
• error: not “TypeError: Failed to fetch” but “Couldn’t refresh the list, showing the saved one. Check your connection and try again”, with a “Try again” button beside it.

Technical details stay in console.error(error). The submit button is switched off (disabled) during the request, or an impatient visitor will press twice and you’ll get two identical inquiries.

Practice

We continue with the 2014 page. We’ll connect it to a “server” and its data. We have no server, of course: the sandbox doesn’t allow real requests. So you’ll paste into the page a block that replaces fetch and answers as the “connection” switch says: all good, slow, lost, server broken, file missing. Your code is real all the same: on a real site you simply delete this block, and the same fetch('/items.json') goes to your server. The block is marked data-scaffold, so it won’t be in the site you download or in the gallery.

Paste the whole block at the end of body, before your own scripts:

  <fieldset data-scaffold id="net-sim">
    <legend>Server simulation: what the connection is like now</legend>
    <label><input type="radio" name="net" value="ok" checked> all good</label>
    <label><input type="radio" name="net" value="slow"> slow, three seconds</label>
    <label><input type="radio" name="net" value="offline"> connection lost</label>
    <label><input type="radio" name="net" value="error500"> the server broke (500)</label>
    <label><input type="radio" name="net" value="missing404"> file missing (404)</label>
  </fieldset>
  <script type="module" data-scaffold>
    // SERVER SIMULATION. The sandbox doesn’t allow real requests, so fetch
    // on this page is replaced: it answers the way chosen above.
    // A real site has no such block, and the addresses lead to your server.
    const snapshot = [...document.querySelectorAll('#offer .card')].map((card) => ({
      title: card.querySelector('h3').textContent,
      text: card.querySelector('p').textContent,
      price: card.querySelector('.price')?.textContent ?? ''
    }));
    const reply = (data, status = 200) =>
      new Response(JSON.stringify(data), { status, headers: { 'Content-Type': 'application/json' } });
    const pause = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

    window.fetch = async (url, options = {}) => {
      const mode = document.querySelector('input[name="net"]:checked').value;
      await pause(mode === 'slow' ? 3000 : 600);
      if (mode === 'offline') throw new TypeError('Failed to fetch');
      if (mode === 'error500') return reply({ error: 'The server broke' }, 500);
      if (mode === 'missing404') return reply({ error: 'Not found' }, 404);
      const path = new URL(url, 'https://site.example/').pathname;
      if (path === '/items.json') {
        return reply([...snapshot, { title: 'A new item from the server', text: 'It isn’t in the markup: it came from items.json.', price: '' }]);
      }
      if (path === '/inquiry' && options.method === 'POST') return reply({ id: 17 }, 201);
      return reply({ error: 'Not found' }, 404);
    };
  </script>

The rest is your part. The list of cards in the markup stays (without a script it must be visible), and when the page opens the script will ask the server for a fresh list and replace the cards with it. The simulation adds one item to the list, “A new item from the server”, so that you can see the data really came from there. The inquiry form is sent without a reload. And play with the connection switch: on “connection lost” and “500” the earlier list must stay in place, with a clear message and a “Try again” button.

If you download the site, put a file called items.json beside it (the sample is above). Until it is there, the page will honestly show the cards saved in the markup and an error message.

You’ll notice one oddity yourself: the quick links from the earlier lessons don’t include the new item. The script built them once, at load, and knows nothing about the list having changed afterwards. It can be fixed, but the more places on a page you must update by hand every time the data changes, the more likely you are to forget one. We’ll come back to this problem in the lesson on frameworks.

What was wrong with the old requests

Before fetch, data was requested with XMLHttpRequest, a clumsy object with callbacks that could also block the page for the length of a request. The technique was called AJAX (the term was coined in 2005 by Jesse James Garrett, describing Gmail and Google Maps): the page updates in pieces, without a reload. The exchange was in XML or JSON, and other people’s sites were requested by a trick called JSONP, which let any script you loaded do anything it liked on your page. Libraries such as jQuery, with its $.ajax, made all of it simpler.

The good part survived: a page that updates in pieces. The bad part: an overloaded page with no indicators that swallows errors silently. A button that does nothing when the network has gone, and doesn’t say so, is still found on sites today.