Asynchronous JavaScript: Understanding Callbacks

Asynchronous JavaScript: Understanding Callbacks

By design, JavaScript is a synchronous scripting language. In fact, many of the widely used asynchronous functions in JavaScript are not part of the core language. Understanding how asynchronous features work in the JavaScript ecosystem, including the role played by external APIs, is an essential part of using the language effectively.

This post will show you how to use asynchronous functionality in JavaScript. It’s also going to explain an important difference between the way code execution works in JavaScript and other languages and frameworks. Understanding this difference is essential to understanding how to do asynchronous programming with JavaScript.

Understanding JavaScript asynchronous code

JavaScript programs rely on a JavaScript runtime environment for execution. Browsers, like Chrome and Firefox, and application servers, like Node.js, include a runtime environment as part of the application. This enables websites to run JavaScript in client browsers and standalone JavaScript applications to run in Node.js.

While Chrome and Node.js use the Google V8 JavaScript engine, Firefox uses its own engine, SpiderMonkey. As part of Microsoft’s re-engineering of the Edge browser, it will also use the V8 JavaScript engine. The two engines differ in how they work internally and produce different performance results in benchmarks, but they can be regarded as substantially the same from the developer’s perspective.

In many execution environments, such as Java application servers, .NET, and Ruby on Rails, speed, scalability, and throughput are handled by spawning new threads of execution. This is how many web servers respond dynamically to changing traffic volume, but it comes with overhead costs in thread scheduling and context switching.

JavaScript uses a single-threaded, nonblocking, event loop to provide concurrency. Rather than rely on thread management to manage multiple I/O tasks, JavaScript engines use events and callbacks to handle asynchronous requests. In the case study code below you'll see the JavaScript event loop in action and you'll get first-hand experience using the callback queue.

The event and callback structure is the fundamental mechanism by which JavaScript engines are able to efficiently handle overlapping tasks, like responding to I/O requests. This is why understanding JavaScript callbacks is essential to understanding asynchronous programming in JavaScript.

In the client browser callbacks enable JavaScript code to make a call that might take a long time to respond, like a web service API call, without “freezing” the web page while it waits for a response. (There’s plenty of bad JavaScript code that does that anyway.)

In server-side applications, like those running on Node.js, it’s the event and callback structure that enables JavaScript to be used effectively for applications that inherently require a lot of asynchronous multitasking, like web servers.

Because JavaScript is a synchronous language by design, and because the event loop is implemented in the JavaScript engines that are part of browsers and application servers, all of the asynchronous functionality in JavaScript is implemented in external libraries. In fact, even commonly used asynchronous functions like setTimeout() are provided by libraries and interfaces.

JavaScript’s reliance on external libraries for I/O activities such as file system and network access originates in the language’s origins as a scripting language for websites. Access to the DOM and functions for manipulating it are built in to the language. As the use of JavaScript expanded into more areas, such as server-side functionality with Node.js, external libraries provided the functionality necessary to perform the I/O and asynchronous functions for those roles.

In this post you’ll learn about basic asynchronous functionality in JavaScript by building your own callbacks, and you'll examine the contents of the event loop. You’ll also see how callbacks can be chained together to execute a sequence of events that manipulate the results of the preceding call when they become available.

Prerequisites

To accomplish the tasks in this post you will need the following:

• Node.js and npm (The Node.js installation will also install npm.)
• Git (if you'd like to use source code control)

You should also have a basic understanding of JavaScript.

There is a companion repository for this post available on GitHub.

Initializing the project

Begin your exploration of concurrency in JavaScript by initializing a new Node.js project. You can initialize the project and its Git repository with a series of command line instructions.

Open a console window and execute the following instructions in the directory where you want to create the project directory:

You can learn more about initializing Node.js projects in this post by Twilio's Phil Nash.

Seeing the JavaScript event loop in action

You can see how the JavaScript event loop works by writing a few simple functions. In the first example the code executes synchronously from the stack and behaves the way you'd expect it to from looking at the code. Then you'll write a function that behaves in a way you might not expect.

Create a greet.js file in the project root directory and insert the following JavaScript code:

Run the program with Node.js by executing the following command-line instruction in the project's root directory:

It’s obvious what the order of the output will be:

You may know the setTimeout() method. It accepts two parameters: the first is a function you want to execute and the second is the time, in milliseconds, the program should wait before executing the function. The function to be executed is passed to setTimeout as an argument. This is a callback function: it's a function you want to run after the function to which you're passing it as an argument does whatever it's supposed to do. In the case of setTimeout, this just means waiting a number of milliseconds.

Modify the greet() function in the greet.js file so it looks like the following:

Run the code:

Your output will look like the following. It's not a bug or mistake.

You were probably expecting exactly the same output as the original greet.js code. The only difference is that you used setTimeout to delay the greet(name) function's console output for one second. So why does the output look reversed? Could it be the 1-second timeout?

Decrease the timeout to 0, as shown below, and run the code again:

You might think that “Hello John!” should be displayed immediately and followed by “Nice to meet you.” Since you changed the setTimeout interval to 0 it seems reasonable that the output from the callback function will appear immediately.

But your results will still be:

This is correct functionality, and it works this way because setTimeout causes the callback function to be executed after a given amount of time—but it does not block execution of the rest of the code on the stack. This is referred to as a "non-blocking" event model, which is one of JavaScript's distinguishing features: it never blocks code on the stack from executing. (There are some exceptions to the single-threaded aspect of JavaScript, but they're beyond the scope of this post.)

The output appears in this unexpected order because of the order of execution in JavaScript's event loop model. Each function in the stack executes completely before execution moves to the next function. You could think about function completion as the moment when it reaches the return statement. (A JavaScript function's return statement is often omitted when the function doesn’t return a value; that's when it's returning void.)

In the greet.js code, the anonymous main function is first in order of execution. When program execution begins the main function is the only function in the stack. As each inner function is called it is pushed to the top of the stack, so its execution completes before execution resumes with the parent function. This can occur with many levels of nesting, which can make a program difficult to debug.

But, as mentioned in the above, JavaScript uses a lot of APIs which are not part of the language. In fact, the setTimeout function is actually part of two external APIs: the Window API, which is included in browser JavaScript runtimes like V8, and the WorkerGlobalScope API that's a part of the Node.js runtime engine.

Here is where the JavaScript callback queue and event loop act as a mechanism to communicate with external APIs. Whenever a function on the stack makes an external API call, JavaScript calls that API and says to it: “Here are the parameters for your task. Do your job, and when you are done leave me a callback function to be executed in the callback queue. I will execute it when the stack is empty”.

In the greet.js code the order of execution works like this:

1. The program begins executing in the anonymous main function.
   1. The greet('John') function call is placed on the stack and begins executing.
      1. the setTimeout(() => console.log(Hello ${name}!), 0); function is placed on the stack and begins executing.
      2. It calls the setTimeout API, which places () => console.log(Hello ${name}!) in the callback queue immediately because the timeout is set to 0.
      3. The saySomethingNice() function executes and displays "Nice to meet you."
   2. The greet('John'); function finishes executing and is removed from the execution stack.
2. The execution stack is now empty, so the event loop checks to see if there are any functions in the callback queue.
3. The event loop picks up the console.log(Hello ${name}!) callback function from the callback queue and places it in the execution stack.
4. console.log(Hello ${name}!`) executes and displays "Hello John!".
5. The program finishes executing.

This is why the messages appear out of order. The functions on the stack, including the one that displays "Nice to meet you", finish executing before functions in the callback queue are executed.

The following illustration shows the event loop described in the sequence of steps above. You can see how the execution of the callback queue relates to the execution of the of the function hierarchy on the stack.

It's essential to keep the event loop in mind when structuring your JavaScript programs. Knowing when callback functions execute in relation to the order of functions on the stack will help you debug your programs.

If you haven't been coding along and you want to catch up to this step using the companion repository on GitHub, execute the following commands in the directory where you’d like to create the project directory:

Understanding callbacks: the key to asynchronous execution in JavaScript

Now that you know how the event loop works, you know how the combination of synchronous JavaScript and the event loop can be used to perform asynchronous execution with external APIs. Using this mechanism you can build JavaScript applications that make use of external APIs that might be unreliable, or take a comparatively long time to return results, without freezing your application while it's waiting for an API.

You can see the process in action by executing a few REST calls using the https://maciejtreder.github.io/asynchronous-javascript/ REST API mockup created for the purpose of testing API clients. You are going to use three endpoints of this API:

/directors returns list of movie directors.

/directors/{id}/movies returns a list of movies made by the specified director.

/movies/{id}/reviews returns a list of reviews for a given movie.

The task which you are going to accomplish is simple: retrieve the top rated movie directed by Quentin Tarantino.

You can simulate the process manually by performing GET calls with your browser:

1. Navigate to https://maciejtreder.github.io/asynchronous-javascript/directors/. You should see list of JSON objects representing directors. Quentin Tarantino has id 2.
2. Navigate to https://maciejtreder.github.io/asynchronous-javascript/directors/2/movies/ to see the list of Tarantino films.
3. See the reviews for Inglourious Basterds by navigating to https://maciejtreder.github.io/asynchronous-javascript/movies/5/reviews.
4. Add up all the scores and divide by the number of scores to get the average review score.

But you are a developer; you don’t want to perform such tasks manually. Building a simple JavaScript application to calculate the average score will demonstrate how the callback queue is used by JavaScript to provide asynchronous functionality.

First you need a way of making HTTP calls from your program to websites. In the post 5 Ways to Make HTTP Requests in Node.js you can find a comparison of some popular libraries used to perform HTTP requests in Node.js.

For this application a simple interface is sufficient, so install the Request library as a new dependency by executing the following command-line instruction in the root directory of your project:

Create a new file, asyncCallback.js, in the root directory of the project and insert the following code:

Run the program by executing the following command-line instruction in the root directory of the project:

As the output you should see detailed list of directors:

Execution of the above code is explained by the following diagram:

1. The program begins executing
2. The request function is placed on the stack and begins executing and launches the Node.js HTTP API through the request library.
3. An HTTP GET request is performed by Node.js on an external web API.
4. The external API returns a list of directors.
5. The  console.log(body); callback function is pushed to the callback queue.
6. The event loop collects the callback function from the callback queue and pushes it to the execution stack.
7. The callback function is executed.
8. The array of directors is displayed in the console window.

Understanding the structure of an external API call using a JavaScript callback

Take a closer look at the code in this function:

The request method, implemented in the request library you added to the project with npm,  accepts three parameters.

The URI of the web API endpoint you want to call:

https://maciejtreder.github.io/asynchronous-javascript/directors

An object informing parser in the request library that the expected response type will be JSON:

{json: true}

A callback function which will be left in the callback queue by the request method when the request is done:

(err, res, body) => {console.log(body);}

The callback function itself also accepts three parameters:

err – a possible error
(If the web API request returns an HTTP 2xx response indicating success, this will be null.)

res – an anonymous object containing the whole web API response including the HTTP headers

body – an anonymous object containing the JSON from the HTTP response body

It's important to note that the callback's parameters, err, res, and body, hold the return values of the web API call and they're used to pass the results of the API call to the function that's placed in the callback queue: console.log(body);

Because the request library contains a JSON parser you don't have to convert the body parameter from JSON to a JavaScript object That's done by the library.

The callback function prints the contents of the body parameter to the console. It's written using the “fat arrow” convention that allows for more compact code. Instead of writing function(param1, param2, param3) {} all that's necessary is (param1, param2, param3) => {}.

Nested callbacks: creating a chain of asynchronous functions

Is it possible to place another asynchronous function call inside a callback? Of course it is! This is what you are going to do to complete the steps you performed manually in your browser at the beginning of the previous section.

Create a new nestedCallbacks.js file in the root directory of the project and insert the following JavaScript code:

Run the program by executing the following command-line instruction in the project root directory:

The output should be:

Examine the code line-by-line. The first call to the external API is similar to the one you wrote in asyncCallback.js, except that instead of returning the list of directors in the body variable it finds the id value for Quentin Tarantino by using the .find method on the directors array and saves the integer result in the tarantinoId variable.

The next request call uses the tarantinoId value to select a list of movies from the /directors/{id}/movies endpoint and store them in the movies parameter of the callback function. 

The callback function for the second web API call is more complex and includes a third API call. For each of the movies in the list returned by the second API call the callback function performs a number of steps:

It uses the id value of the movie object to get a list of reviews for the movie using a call to the /movies/{id}/reviews endpoint.

The review.rating from each review returned by the third API call is added to the aggregatedScore for the movie.

The movie.averageRating is calculated. (Because this is JavaScript you can add a property to an object on the fly. No need to worry about the data type, either.)

If the number in checkedMoviesCount equals the number of movies, as determined by getting the length of the movies array, the code sorts the list of movies in descending by averageRating and writes the title of the movie to the console window as part of the output identifying the director's best movie.

You've performed a complex series of data queries from a remote source with about 20 lines of code. As you can see, you can perform quite a bit of processing in your callback functions as well.

It's only because the callback queue is processed in order and one callback doesn't commence before the next one finishes that this series of nested callbacks produces predictable results. Waiting for the results wouldn't prevent other code in the application from executing because the callback functions in the callback queue wouldn't block other functions in the stack from executing.

If you haven't been following along and want to catch up to this step using the companion repository on GitHub, execute the following commands in the directory where you’d like to create the project directory:

Taking the next step in asynchronous programming with JavaScript callbacks

As you can see, using the results of a callback in a series of callbacks is a powerful technique. But in more complex scenarios it can make your code difficult to read and debug. Too much nesting can also make it difficult to determine where you are in the callback queue.

In the next post in this series on asynchronous JavaScript you'll learn how to structure your code so your callbacks are easier to read and debug. By doing so you'll also be make your callbacks easier to reuse.

Summary of Understanding JavaScript Callbacks

In this post you took a deep look inside the heart of JavaScript at the event loop and callback queue. You also learned how JavaScript runtime environments can simulate multi-threaded execution with asynchronous calls.

Additional resources

What is Node.js? The JavaScript runtime explained

V8 JavaScript runtime engine documentation

SpiderMonkey JavaScript runtime engine documentation

Microsoft Edge: Making the web better through more open source collaboration – Microsoft's blog post announcing the switch to the Chromium and V8 engines, known as "Credge"

ECMAScript (JavaScript) compatibility table

Maciej Treder is a Senior Software Development Engineer at Akamai Technologies. He is also an international conference speaker and the author of @ng-toolkit, an open source toolkit for building Angular progressive web apps (PWAs), serverless apps, and Angular Universal apps. Check out the repo to learn more about the toolkit, contribute, and support the project. You can learn more about the author at https://www.maciejtreder.com. You can also contact him at: contact@maciejtreder.com or @maciejtreder on GitHub, Twitter, StackOverflow, and LinkedIn.