Sometimes the data you need is available online, but not through a dedicated REST API. Luckily for JavaScript developers, there are a variety of tools available in Node.js for scraping and parsing data directly from websites to use in your projects and applications.

Let's walk through 4 of these libraries to see how they work and how they compare to each other.

Make sure you have up to date versions of Node.js (at least 12.0.0) and npm installed on your machine. Run the terminal command in the directory where you want your code to live:

For some of these applications, we'll be using the Got library for making HTTP requests, so install that with this command in the same directory:

Let's try finding all of the links to unique MIDI files on this web page from the Video Game Music Archive with a …

Sometimes the data you need is available online, but not through a public API. Web scraping can be useful in these situations, but only if this data can be accessed statically on a web page. Fortunately for developers everywhere, most things that you can do manually in the browser can be done using Playwright, a Node library built by the same team that made Puppeteer which provides a high-level API for automating various browsers.

Let's walk through how to use Playwright to interact with web pages programmatically. In this example we'll use the Native Land Digital tool, an awesome project built to teach people more about their local indigenous history. In this case, an API does exist, but it only takes location data in the form of geo-coordinates rather than a more user-friendly address. We'll write code to programmatically type an address and figure out which Native land corresponds …

The internet has a wide variety of information for human consumption. But this data is often difficult to access programmatically if it doesn't come in the form of a dedicated REST API. With Node.js tools like jsdom, you can scrape and parse this data directly from web pages to use for your projects and applications.

Let's use the example of needing MIDI data to train a neural network that can generate classic Nintendo-sounding music. In order to do this, we'll need a set of MIDI music from old Nintendo games. Using jsdom we can scrape this data from the Video Game Music Archive.

Getting started and setting up dependencies

Before moving on, you will need to make sure you have an up to date version of Node.js and npm installed.

Navigate to the directory where you want this code to live and run the following …

Around the world, the Coronavirus Pandemic is affecting people and their communities. It might impact different areas in varying ways, but the virus causing the COVID-19 disease has reached a level of global pervasiveness. The situation is constantly evolving so it can be hard to keep track of.

If you are in the United States, there is a COVID Tracking Project which provides up-to-date statistics for the nation as a whole and for each individual State on the number of tests administered, positive cases, and the death toll. The project also provides this data in the form of an API.
You can text the phone number +1 (864) 77-STATS which is +1 (864) 777-8287 with a particular US State abbreviation to get the current statistics for that State, or any other message to get the nation-wide data.

Let's walk through how to build a text-message-based COVID-19 tracker in Python …

Another conference call, another app, another PIN, another log-in. Joining conference calls shouldn’t take more than simply dialing a phone number. Lets walk through how you can quickly build a conference line anyone and everyone can use with Twilio Functions.

Setting up a Twilio Function

You’ll need to create a Twilio account if you don’t already have one. From the Twilio Functions page in your Twilio Console, create a new Function and select the “Blank” template as seen in this image:

Give your function a name and a path.

Writing code to handle incoming calls

Now replace the default code in the function with the following code for our app, and click save:

Another conference call, another app, another PIN, another log-in. Joining conference calls shouldn’t take more than simply dialing a phone number. Lets walk through how you can quickly build a conference line anyone and everyone can use with Node.js.

Development Environment Setup

Start by making sure you have the right software installed that you'll need to use for the rest of this post. You will need:

• Node.js and npm installed (do this first if you haven't already)
• A free Twilio account and a Twilio phone number
• The npm modules for Express and the Twilio Node library

Here is a good guide to follow in general if you are going to be doing more with Twilio and Node.js and have any more questions.

To install these npm modules, navigate to the directory where you want this code to live and run the following command in your terminal to create an npm …

Making HTTP requests is core functionality for modern languages and one of the first things many developers learn when acclimating to new environments. When it comes to Node.js there are a fair amount of solutions to this problem both built into the language and by the community. Let’s take a look at some of the most popular ones.

A few years ago, I wrote a similar post on this topic. But now that async/await functionality is more pervasive and mainstream in JavaScript code, making network requests is more straightforward than ever. On top of that, Request, the previously most popular HTTP library for Node has been deprecated. So it's time for an updated guide!

As in the other post, we’ll be using NASA’s Astronomy Picture of the Day API as the JSON API that we are interacting with in all of these examples because space is the coolest …

Sometimes the data you need is available online, but not through a public API. Web scraping can be useful in these situations, but only if this data can be accessed statically on a web page. Fortunately for developers everywhere, most things that you can do manually in the browser can be done using Puppeteer, a Node library which provides a high-level API to control Chrome or Chromium over the DevTools protocol.

Let's walk through how to use Puppeteer to write scripts to interact with web pages programmatically. In this example we'll use the Native Land Digital tool, an awesome project built to help people learn more about their local indigenous history. In this case, an API does exist, but it only takes location data in the form of geo-coordinates rather than a more user-friendly address. We'll write code to programmatically type in an address and figure out which …

The internet has a wide variety of information for human consumption. But this data is often difficult to access programmatically if it doesn't come in the form of a dedicated REST API. With Node.js tools like Cheerio, you can scrape and parse this data directly from web pages to use for your projects and applications.

Let's use the example of scraping MIDI data to train a neural network that can generate classic Nintendo-sounding music. In order to do this, we'll need a set of music from old Nintendo games. Using Cheerio we can scrape this data from the Video Game Music Archive.

Getting started and setting up dependencies

Before moving on, you will need to make sure you have an up to date version of Node.js and npm installed.

Navigate to the directory where you want this code to live and run the following command …