Why not hack your news? A simple guide to using the New York Times API in Python

Whether you’re looking for an interesting research project or are just tired of scrolling — this tool is exactly what you need

The New York Times is one of the most renowned and trusted newspapers worldwide. It is an excellent resource to stay informed on the latest in politics, skim movie reviews, or try your luck doing some natural language processing on their business news.

What if I told you that you could get all this data from the New York Times website using the New York Times API? Here’s how to do it.

When you have created your account, click on your email-address at the top right-hand side of the screen and select “Apps” in the drop-down menu. Now, select the blue “+NEW APP” icon and give your new app a name.

Two sections are important to mention. Firstly, scroll all the way down to the APIs * section. You can see here that the New York Times API has a number of different features that you can enable or disable for your app: data about the top stories, the most viewed or shared articles, book and movie reviews as well as bestseller lists, article metadata or the NYT Archive. For the purpose of demonstration, I will only focus on the Article Search feature in this article. If you would like to follow along with this tutorial, make sure you have this feature enabled.

Once you have enabled the Article Search feature, go back to the API KEY section and copy your API key.

When you have pynytimes installed, you need to import the module. You can do this by typing the following at the top of your Python Script:

Now, you need to set up your API. Remember the API key we created in the previous section? Copy your key, and insert it into your code (the brackets are just meant as a placeholder — do not use them in your code):

Congratulations! You have now successfully set up the New York Times API! That was easy enough, wasn’t it?

Now the fun part can start. With the Article Search API, we can search all of the currently listed articles on the New York Times website and filter by keyword, title, publication date, location, source, news desk and many more. Let’s start by building a simple query to extract all of the opinion pieces containing the keyword “climate change” that were published in 2020:

We have now successfully retrieved our articles of interest from the website. However, the output is a combination of nested lists and dictionaries — still somewhat cryptic and not very user-friendly. It also includes a lot of data we are not interested in. In this section, I will show how to retrieve the information we want, store them as a pandas data frame, and export them in an excel or CSV format.

Let’s have a look at the structure of the query output. It is a big list containing multiple — in some cases nested — dictionaries. To extract relevant information, I use the map-function. This function allows me to execute a given function on any element of a list. This is an excellent feature to retrieve information from nested dictionaries or lists. The lambda-function retrieves the dictionary value for the elements of the query output I am interested in:

The map-function returns a map object. To feed the data into a data frame, we still need to convert them into lists:

The data can also be exported into a format of choice:

Related posts:

This post is part of the D as Diagramming project which aims to explore the power of diagrams and diagramming. What I really want to know is about the value of diagrams for turning tacit knowledge…

The keplerSwap SPACE enable users to invite more people to participate in community advocacy so as to provide more support for development of the ecosystem. SPACE is a term used in the KeplerSwap…

Cryptocurrency has been on a huge run this year. Cryptocurrency gambling also has been gaining increasing popularity online. As our world economy grows, assets like cryptocurrency are fuelled by big…