United States 117th Congress: A Preliminary Data Exploration

Introduction

For a long time, I’ve wanted to interact with legislation data. I’ve been curious about the process of how a bill becomes a law, and what kind of bills are introduced versus what kind of bills are passed. More than anything else, I’ve been curious about how one could take such data and find ways of exploring and presenting it to people such that they could understand the process better themselves. Add a little transparency to the process, if you will.

Well, I’ve finally gotten around to this project, and today I want to discuss the data I’ve collected and how I’ve gone about exploring it. In the future, I want to drill down into geographic analysis of the data (i.e. what states are introducing what bills, what states are passing what bills, etc.) at the federal level. I also want to see if there’s ways one could observe the politicians themselves and develop a model for predicting how they will vote on a bill.

All of these goals are quite ambitious, so I’m going to start small. Today, I’ll be focusing on the data I’ve collected and some basic exploratory analysis.

Data - How It Was Obtained

My primary data source is Congress.gov. This site is maintained by the Library of Congress and is a great resource for anyone interested in the legislative process.

There’s way more data than I could possibly use, so for now I’m focusing on the 117th Congress. Furthermore, I’m only considering resolutions and joint resolutions, but in the future we may consider concurrent resolutions as well. We omit all amendments for the time being.

Here’s the data I collected:

We’ll dig in a bit deeper into the data in the next section, but for now I want to discuss how I obtained the data.

Funny enough, building out a crawler for this site was pretty easy. The site is well organized and the data is presented in a consistent manner. I used the following technologies:

• Python
• BeautifulSoup
• Selenium
• A little bit of regex magic

Because the site dynamically loads data, I had to use Selenium to load the page and then use BeautifulSoup to parse the HTML instead of something like requests. Furthermore, to respect the site’s bandwidth, I added a 5-second delay between each page load. This resulted in the crawler taking about 3 days to run.

The crawler is available on GitHub if you’re interested in checking it out. In the near term future, I’ll also release the pre-processed data publicly so that others may be able to use it and provide their own analysis. I’ll also briefly summarize the process of crawling the site here.

For every bill, we need to query two pages:

• The bill’s all info page, containing everything but text: f'https://www.congress.gov/bill/{congress}th-congress/{bill_type}/{bill_id}/all-info'
• The bill’s text page, containing the current text of the bill: f'https://www.congress.gov/bill/{congress}th-congress/{bill_type}/{bill_id}/text{version}?format=txt'

You’ll notice that there are three variables in the URLs: congress, bill_type, and bill_id. For this analysis, we only consider congress=117 (the 117th Congress, 2021-2023). We also only consider bill_type in the set:

• house-resolution
• house-joint-resolution
• senate-resolution

Finally, we only consider bill_id in the set of all bill IDs for the given bill_type and congress. For this, I just manually checked the last bill ID for each bill_type and congress and then iterate from 1 to that number.

I implemented a basic caching mechanism to avoid re-querying the same page multiple times, that way I could re-iterate my post-processing code without having to re-query the site.

Actually parsing the HTML was pretty straightforward. Just a matter of targeting key divs, tables, and spans. Here and there, I use a dash of regex for simple post-processing of text.

Data - What It Looks Like

In this post, I want to focus on all the top-level features I’ve collected from the data. This means that, for now, I won’t be crossing features to see their joint distributions, I’ll also be ignoring the text of the bills for now. The rest of this article will be partitioned into sections based on the feature.

Overview - Tracker Status

First, I want to focus on what people probably care most about. What bills are being introduced and what bills are being passed? What happens to a bill in a given congress?

For every bill that ends up on Congress.gov, there’s a tracker status that indicates where the bill is in the legislative process. Based on the statuses in this current data set, the tracker status is a string that can take on one of the following values:

If we like, we could further collapse these categories into three coarse categories:

• Introduced: Any bill that has been introduced to a chamber but never seen a vote (i.e. Introduced)
• Stalled: Any bill that has seen a vote but has not become law. Especially since the session is over, we can assume that any bill in this category will not become law (i.e. Passed House, Passed Senate, Failed House, Failed Senate, Resolving Differences)
• Law: Any bill that has become law (i.e. Became Law, Became Private Law)

Some quick observations here:

• Less than 7% of bills introduced ever see a vote. They’re either introduced and then forgotten about or they’re introduced and then die in committee.
• Of the bills that see a vote, around 36% become law. Compared to the total, approximately 2% of bills introduced become law.

All of this is conditional on the bills I’ve collected, and of the 117th Congress.

Overview - Sponsor

The sponsor of a bill is the primary member of Congress who introduced the bill. Sponsors have a party affiliation and a state, so we can look at the distribution of bills by party and state. This assumes those factors are relevant to the bill’s success, which may or may not be true.

Party

Some observations:

• Democrats:
  • Around 7.5% of bills introduced by Democrats moved out of the Introduced stage.
  • Around 2.6% of bills introduced by Democrats became law.
  • Around 35% of bills that moved out of the Introduced stage became law.
• Republicans:
  • Around 5.5% of bills introduced by Republicans moved out of the Introduced stage.
  • Around 2.1% of bills introduced by Republicans became law.
  • Around 38% of bills that moved out of the Introduced stage became law.

So, overall, Democrats are introducing more bills and more of their bills are becoming law. However, the percentage of bills that become law is fairly similar between the two parties. Additionally, Republicans have a higher percentage of bills that move out of the Introduced stage and become law.

State

Here’s a table of the top 10 states by number of bills introduced:

We might also want to normalize by the number of representatives per state. This would give us a better idea of which states are introducing the most bills per representative.

Here’s a table of the top 10 states by number of bills introduced per representative:

Individual

Why not also look at the top 10 individual sponsors by number of bills introduced, etc.?

What about a score based on a ratio of bills introduced to bills that became law?

$$ \text{score} = \frac{\text{bills that became law}}{\text{bills introduced} + \text{bills stalled} + \text{bills that became law}} $$

Policy Areas

Every bill is assigned a primary policy area. Here, we’re just doing a quick look at the most common policy areas by bill status.

Wrapping Up

This is just a quick look at the data, and there’s so much more to consider. We haven’t looked at committees or cosponsors. We haven’t looked at voting patterns or the text of the bills themselves. Nevertheless, we can see that there are some interesting patterns in the data, and we can use this to guide our future analysis. Hopefully these tidbits will inspire you to dig deeper into the data and find your own interesting patterns.

If you have any thoughts, questions, feedback, or general concerns, don’t hesitate to reach out!