A summer of live streamed coding

For the past few months, Ian and I have been experimenting with livestreaming ourselves working on dataviz projects on Twitch and calling it Coding With Fire.

These experiments evolved from the rough shape of how Ian and I have hung out since we met nearly 10 years ago as coworkers at a dataviz startup in San Francisco. During coffee breaks and after work we would hang out and hack on projects that didn't fit into our day jobs.

It has been interesting to take that previously private pattern of collaboration and do it in public. I think it made it easier to work through some of the other challenging aspects of livestreaming. We already had a rapport and an understanding of how to work together that allowed us to multi-thread an idea, work at it separately, and combine our individual elements later.

I think it would be much more difficult to attempt to do the same thing alone, and I am glad to have a skilled co-host to work with.

A lot of inspiration came from other livestreamers who are finding new methods of creating entertainment in a world facing drastically new paradigms due to a global pandemic. I have enjoyed watching people build keyboards, play video games, and code on Twitch.

It has also been really inspiring to see the discipline and focus of Shirley Wu as she builds a community around her data exploration and visualization livestreams.

I think that the more people who understand the skills required to visualize data, the more people who are exposing truths hidden in datasets, the better the world will be. I also hope helping people who want to get into data visualization see the process will be valuable in terms of education as well as inspiration.

Table of contents

The cooking show metaphor

One thing that really helped us figure out how to think about how to present ourselves was the metaphor of a cooking show. I grew up watching all sorts of excellent and varied cooking shows; Masterchef, Kitchen Nightmares, Iron Chef, Chopped, Top Chef, and more.

The thing I like about these cooking shows is the same thing I like about shows like Mythbusters; I got to see someone make something every episode.

This also helps me think of my show preparation the same way a chef might prepare for a service. I do my mise en place and get all my ingredients sliced and diced in the right proportions. When it comes time to do the show, I am left with the entertaining parts of assembling them, talking about why, and experimenting.

Too many modern educational coding videos fall closer on the spectrum to watching a chef cut their carrots individually, and that's not the type of TV show that I wanted to create.

The cooking shows that I love are more than someone following a certain recipe. I love when a chef lets me into their decision making process, the sort of meta-thinking and skills that allow someone not only to imitate the immediate actions of a skilled chef but also adopt reusable formulas that can be used in yet-unforeseen circumstances.

Working in the open

Another great benefit of livestreaming yourself working is that there is a flow of education back and forth between the audience and the hosts.

Feedback can range from "you missed a letter there" to "try out this new library". It is really motivating to have people helping you along in your journey. It makes it feel like you are with a group of people solving interesting problems together (which reminds me of my favorite moments when working at start-ups).

You also get a record of the decisions you made, which I have used on more than one occasion to remind myself why I did something a certain way.

Livestream experiments

Live animation

One time, Ian and I turned ourselves into animated characters using Adobe Character Animator. I thought it turned out awesome.

For me, this is partially solving the problem of what visual to show for audio-only guests.

It also opens possibilities for other experimentation with presentation, like if our animated characters occupied a virtual space, or acted out a narrative in real time in combination with the code we were writing.

Mozilla Hubs

While I haven't used it much yet, I am curious about hosting livestreams using Mozilla Hubs which is a cool way to spatialize guests, reference material, and outputs. I am imagining some way of putting a bunch of screenshots on a shared table, going over them together. Then, in the virtual space, we might go to our "laptops" and start live coding. Our outputs are displayed in the virtual world on enormous screens (or floating in space, if 3D), and users can join in and explore them and offer feedback.

Crowd participation through Firebase

Ian created a notebook that uses Firebase and a real-time database so that anyone can edit the colors that appear behind us live on stream. The colors are persistent, and anyone can change them, so we often start the stream with a cool new color scheme created by an anonymous person.

I love every opportunity to have this type of collaboration with the audience. I also really enjoy the parameters of collaboration we have set. A malicious troll could potentially make the colors all white, or something, but even that is interesting.

What they can't do is edit our color palette in a way that ends up with us saying something racist, misogynist, or otherwise unwanted on our stream.

Letting the crowd control the colors has been such a rewarding and risk-free way to let the users interact with us directly and see the changes they are making in real time on the stream (when the background is showing).

Building the habit

One of the best parts about doing the livestream with Ian is that we can hold each other accountable. Skipping a stream also means letting your friend down. As a result of this, the only streams we have missed happened when Ian had to move his entire family across the country from California to Florida in the middle of a pandemic.

We do our streams on Sunday and Monday evenings every week, the repetition has been the key to figuring things out. Different technical and graphical elements emerged as we found that we needed them. We went through the occasional technical hiccup on air. We did a lot of streaming to 1 or 2 viewers and figuring out how pre-produce the shows, pace ourselves, estimate how long different tasks would take, and know when to move on when something was going more slowly than expected.

Surprisingly productive

One of the most surprising things of working in the open is that I feel like every time we stream we are surprisingly productive. There is something unique about "performing" your job to the internet. We usually stream for 3 hours and take a little 10-15 break in the middle. I rarely get that sort of consistently focused time in my private life. Because of that, I find myself waking up the morning after a livestream and looking at all the things that we created in astonishment.

"We did all that?" I ask myself, but of course we did, it's all on video.

Online Nazis Have Figured Out How To Game Twitter

One of the newest weaknesses online trolls are exploiting is one of the mechanisms built to defend against them, Twitter’s system for reporting tweets that violate their Terms of Service.

Twitter doesn’t detail publicly the mechanisms used to decide which tweets and accounts get removed, how many reports trigger a human moderator, or whether a certain amount of reports automatically triggers a ban.

Recently the Twitter account for the magazine Jewish Worker @JewishWorker was suspended from Twitter for “posting private information” after a series of tweets about a journalism project that revealed the identities of alleged white supremacists.

Another victim of targeted mass reporting is the journalist Barrett Brown who was repeatedly banned from Twitter after being targeted by a group of far-right trolls known to organize and mass-report their online enemies, and who publicly celebrated their success in gaming Twitter’s moderation systems and getting Brown banned.

Telegram groups, like the “Mass Report and Retweet thread” previously reported in March by the Daily Beast continues to be filled with one-way blasts of Twitter posts, accompanied by which option should be selected when reporting the tweet: most often “report as abusive”, or “personal info”.

A tweet with photos of 4 alleged Proud Boys was posted to the channel along with the text “targeted harrasment”, indicating to members of the channel what they should report the tweet for in order to trigger a takedown.

Some of the links to tweets they target for mass-reporting have been deleted, meaning they were removed by Twitter or the users themselves.

The Proud Boys Telegram channel also celebrates when they succeed in gaming the Twitter moderation system and getting an enemy suspended. – after an independent journalist got the attention of the Proud Boys, she was targeted. A screenshot of her suspension was later posted to the Telegram channel accompanied by the text “Great job! Let’s keep this shit up!”

Even in cases where mass-reporting was unable to trigger automated or semi-automated systems within Twitter to suspend the targeted accounts, the barrage of attention and abuse from the organized trolls had the desired effect.

One post from the group celebrates a target deleting their own post: “Good work all around boys, we are calling it off now. The leftist in question has removed his post attacking our brothers business due to being corrected by all of you.”

Activists have responded with criticism of Twitter, with one user tweeting “The message from Twitter is clear: If a group of violent white supremacists mobilizes to silence journalists for exposing their allies' identities, they will comply with the wishes of the violent white supremacists.”

Things I've learned about elections

Edge cases

Elections are all edge cases. Votes are cast and tallied by a wide variety of Americans who all have the same goal but often different ideas about how to go about it. Votes are cast in cities and rural areas, in places with millions of constituents or dozens. Many elections are run locally by someone you should mentally picture as your well-meaning aunt. The sheer scale of such an operation is astounding. We live in a country that spans a continent and 9 time zones.

This great country contains lots of different people with lots of different ways of going about things.

Table of contents

The basic rules around voting are laid out in the constitution^[Article 1 of the Constitution “The House of Representatives shall be composed of Members chosen every second Year by the People of the several States…” - Section 4: “The Times, Places and Manner of holding Elections for Senators and Representatives, shall be prescribed in each State by the Legislature thereof; but the Congress may at any time by Law make or alter such Regulations, except as to the Places of chusing Senators.”] but a remarkable amount of the nitty-gritty details are left up to each state to figure out.

On election day in America, polls open at a particular time and close at a particular time. Those times, like almost everything else, depend on where you’re talking about and can vary even within a single state and could change any year.

We have grown accustomed to live results of elections. This is definitely not something our grandfathers had and is mind-bogglingly complex once you start to think about it.

With American elections “live” does not mean getting the results of votes immediately after they are cast. The closest we can get for now is seeing vote results as they are counted after polls close. ^This makes sense, when you think about it. There is reason to believe if you could turn on your TV and see how your candidate was doing, if you could see the vote counts live and your candidate was safely ahead, you might decide it’s not even worth going out and casting a vote.

Races

Any particular office (be it a Senate or a President or local Sheriff) has a variety of candidates from a variety of parties. Generally we think of races as between two front runners, but people often vote for third parties or write in candidate’s names and those votes need to be accounted for.

Here’s what we can agree on: A race is run by n number of candidates for a single office.

You might say "at least we can assume the candidates are human" - but you would be wrong. Never assume in elections.

Every eligible(!) person who lives in the area that candidate represents (districts for the house, states for senators, the entire country for presidents) gets to vote between the candidates running for each office that represents them. ^The election of Senators was not always given to the population in general. The framer’s constitution left that to the states. “The framers believed that in electing senators, state legislatures would cement their ties with the national government. They also expected that senators elected by state legislatures would be freed from pressures of public opinion and therefore better able to concentrate on legislative business and serve the needs of each state. In essence, senators would serve as “states’ ambassadors” to the federal government.” - it turned out to not work well because states never got around to electing them.

Voting

Constituents show up to their polling place. Or, well, sort of. 39 states have mechanisms for early voting, 28 allow no-excuse absentee votes, 19 allow voting with an “accepted excuse”. 36 million people voted early in the midterm election. - that’s about 30% of the 117 million total votes cast.

When reporting votes, absentee votes are typically only available at the county-level and not be precinct.

Voter Registration

In order to vote, an eligible potential voter must register. Voter registration is handled by government officials on the county and state level.

Once registered on voter records, voters can be removed for any of the following reason: Voters can be removed from records for a few reasons:

• Disenfranchising criminal conviction
• Mental incapacity
• Death
• Change in residence
• Never eligible: under 18 or noncitizen
• Voter self-requests removal

EAVS (Election Administration and Voting Survey)

EAVS is a report created from a survey of every state in the US with information about how elections are carried out.

Voting systems

• First past the post is what most people think of. Voters mark a ballot with the candidate they want to win, and the candidate who receives the most votes wins.
• Runoff voting works similarly, but there is a minimum amount of votes required. If no candidate exceeds that minimum, some candidates are eliminated and a second round of the election is held (usually with the top 2 ranked candidates).
• A Louisiana primary is similar to an open primary where all candidates running for an office appear together on the same ballot, including multiples per party (instead of a party picking a single candidate to run in a race). The race goes to runoff if no candidate wins by simple majority.
• Maine is the first U.S. state to enact instant-runoff or “ranked choice” voting^A variety of major U.S. cities use ranked choice voting for local elections. Instead of voting for one candidate over another, voters rank the candidates in order of preference.

The Center for Civic Design has a great guide for designing ballots so they are usable, which they are often not.

Reporting units

The term “reporting unit” can refer to a boundary division of any size used to aggregate votes in an election. This can be a town, parish, precinct, county, county sub-division or state.

Presidential, senate, and gubernatorial races are organized by county and state.

House races are organized by house districts. The borders of these house districts are primarily decided by state legislatures but some (AZ, CA, ID, WA) are made by independent commissions. 538 did an excellent piece on congressional redistricting and gerrymandering.

Election data

Once a vote is cast, ballots are counted and the counts are delivered to a central location and the numbers are reported in regular intervals (usually). That is all handled by state and local governments, often by county and state boards of elections.

The media (and by extension, the public) are given access to the vote counts at this point. Often times you can find a website run by the Secretary of State which will have the latest results for different statewide elections. Data services will pay reporters or stringers to go to the places where votes are counted to and send the results back to competing data services which then resell that data to news organizations (live election results are not cheap).

Election day vote and exit poll result data is provided to ABC, CBS, CNN, and NBC News in a consortium called the National Election Pool. The data is fed by Edison Research.

Alternatives for live data include Decision Desk HQ or the AP Elections API.

Historical data can be found from the above sources as well as from MIT or OpenElections. MapBox also has pre-bound boundaries with historical data.

County-level presidential results 2008-2016 have been collected in this GitHub repo

Geographic Identifiers: FIPS Codes

States can be identified via their name (New York), abbreviation (NY), or a state FIPS code (36).

House districts are identified by their district code. New York’s first district is NY-01. Large states with smaller populations like Montana have only one congressional district (“at large”), denoted like MT-AL.

I’ve seen various systems handle this different ways. Some designate these districts with MT-00 so at least you can rely on two numeric digits when writing your code.

Counties are identified by county FIPS codes. The first two numbers of a county FIPS code are the state code, which is handy.

Mapping live election results

At NBC the Big Board is commanded by Steve Kornacki, who is the best in the business if you ask me. He stands by an enormous touch screen showing the latest vote results and explains what is happening and why it is interesting.

Typically he is pointing at a map of the country, or a particular state. That map has various counties or districts or shapes colored red and blue. ^During my time at NBC, the rules about when to color shapes has been debated. House districts are only colored for a particular party when that race has been called (not projected) by the NBC decision desk. This means there is basically no chance it will ever change color. Counties, on the other hand, never technically have a “winner” but a “leader” who has a majority of votes in a particular county. Counties were colored by whoever was leading the votes. Once the first votes came in, if there were 50 for the D and 51 for the R, the county would color for the R.

The election data comes in as a big list of states, counties, and districts. A handy way to refer to all of these buckets of votes at once is as a reporting unit.

Each reporting unit contains different races.

Primary elections

In addition to the general elections that gather the most attention, parties hold their own elections to decide who will run against the other party, the primaries.

These primaries can take various forms including: uncontested (no challenger) or runoff/Louisiana/blanket. They can be open or closed, partially open, partially closed, etc… - remember, edge cases.

Voting machines

Voting machine^There are 4 main types of voting machines: Optical Scan Paper Ballot (little bubbles you fill out by hand), Direct Recording Electronic (computers with touch screens or buttons), Ballot Marking (a computer marks paper), and Punch Card (voters punch holes and avoid whatever “hanging chads” are) security in the United States is currently a joke.^See the reports from the DEFCON Voting Machine Hacking Village in 2018 and 2017 The joke is not funny when you consider the real-world impact of such an important lever of our society left comically unguarded.

The OSET Institute has compiled some data and an excellent team to work on the technology used for American elections, including defending from threats of cyberterrorism.

Coverage of elections

In 2018 I did some design research (captured in a twitter thread) and went through some old footage of NBC election coverage. I was even able to incorporate the effect of numbers counting up as they changed into the 2018 version of the Big Board.

I had the fortune to join a politics-covering media organization in 2016, a year when suddenly there was intense interest in how media organizations covered politics. Many feel the media failed in the coverage of the 2016 election.

I think it is worthwhile to listen to those criticisms and think of ways that things might be improved.

But we have a rat’s nests of problems to untangle from horse race coverage (which might be good!) to televised debates, election polling methodologies, and the ways polls should be covered.

Horse race election coverage

Jack Shafer wrote in Politico about why horse race election coverage might be good.

By giving voters a window on the closed world of insider politics, horse-race stories help focus reader attention on the races. Without the work of election handicappers, coverage would come to resemble an endless series of policy white papers that nobody reads. The presidential campaign has another 22 months to run, leaving plenty of time and space to explore the contest from multiple perspectives.

Election night 2000

At NBC the old heads would whisper to never fuck up as bad as they did back in 2000. A CNN internal report called their own election night coverage in 2000 “a disaster”.

Following the election night coverage – in which CNN, ABC, CBS, NBC, Fox and MSNBC all made wrong calls on the Florida vote – several networks ordered investigations of their coverage.

Based largely on exit polling, the networks first declared Mr. Gore the winner in Florida, only to give the state to Mr. Bush later. Ultimately the networks backed off again, declaring the race undecided.

The coverage of 2000’s presidential election lead to executives from CBS, CNN, Fox, NBC, and the AP being called to testify in front of congress by committee chair “Billy” Tauzin (R-La.).

Much of the testimony focused on the networks’ reliance on exit polls and the flaws in the VNS system of conducting the polls.

Tauzin introduced legislation to create a uniform poll closing time across the country and asked the networks to voluntarily resist projecting winners until 9 p.m. EST. In a concession, all the networks said they will refrain from calling a state until all the polls in the particular state have closed.

Why do we have elections on Tuesdays?

It could be argued that it would make more sense for elections to be held on the weekend.

Erin McCarthy wrote for Mentalfloss:

Monday was out, because it would require people to travel to the polls by buggy on the Sunday Sabbath. Wednesday was also not an option, because it was market day, and farmers wouldn’t be able to make it to the polls. So it was decided that Tuesday would be the day that Americans would vote in elections, and in 1845, Congress passed a law.

Resources

• Ballotpedia is an exceptional source of all of the strange little constantly-changing details about elections, --freely editable like Wikipedia-- edited by Ballotpedia staff.
• election-geodata is a great collection of geographic precinct shape data for mapping election results.
• OpenElections is an open-source project made to create a “free, comprehensive, standardized, linked set of election data for the United States, including federal and statewide offices” which is a pretty incredible goal if you ask me.
• 270toWin has great maps and information on historical elections.