Edward Snowden on Privacy, Values, and Government Surveillance

John Oliver’s recent interview with Edward Snowden is circulating widely online, but given the ignorance displayed in “man on the street” segments it deserves more attention. Those segments are included in the video below, and misapprehensions range from the belief that Snowden sold secrets to associating him with WikiLeaks (that’s not Snowden, it’s Julian Assange).

To be clear, this is not a sympathetic interview. There are some humorous parts, but Oliver is not tossing softballs. He asks Snowden difficult questions about whether he understood the implications of the information he released, and whether turning it over to journalists was a responsible decision. As a Snowden supporter some parts of the conversation were frustrating to me, but that probably means that it was a fair interview.

The most important part of the video below concerns Section 215 of the Patriot Act, which allows government collection of “any tangible thing” that is “relevant to” a terrorism investigation. That’s an unbelievably broad provision, and one that its author James Sensenbrenner claims has been misinterpreted by lenient FISA courts and should be put out of business. We must have a very serious national conversation about this bill before its renewal on June 1 of this year.

There were several good quotes from Edward Snowden in the interview. Here are my two favorites:

You will never be completely free from risk if you’re free.

You shouldn’t change your behavior because a government agency somewhere is doing the wrong thing. If we sacrifice our values because we’re afraid, we don’t care about those values very much.

The New Netflix Strategy: Gambling on House of Cards

NetflixGamblingOne week ago Netflix introduced its first original series, House of Cards. The series details the life and crimes of (fictional) US Congressman Francis Underwood and his wife Claire who runs a nonprofit. What is unique about the series is that the entire season–13 episodes–was released all at once. Netflix and streaming services like it have acclimated us to watching shows in bulk like this. Is the new model sustainable?

I hope so, and Atlantic Wire reporter Rebecca Greenfield thinks the answer is yes:

With Netflix spending a reported $100 million to produce two 13-episode seasons of House of Cards, they need 520,834 people to sign up for a $7.99 subscription for two years to break even. To do that five times every year, then, the streaming TV site would have to sign up more 2.6 million subscribers than they would have. That sounds daunting, but at the moment, Netflix has 33.3 million subscribers, so this is an increase of less than 10 percent on their current customer base. Of course, looking at Netflix’s past growth, that represents pretty reasonable growth for the company that saw 65 percent growth from 20 million to over 33 million world-wide streaming customers. Much of that growth, however, comes from new overseas markets. But, even in the U.S., from one year ago, Netflix saw about 13 percent streaming viewer growth jumping from 24 million to 27 million.

The five times per year figure comes from a plan that Netflix CEO Reid Hastings revealed in an interview with GQ. Paying for subscription television like this is not a new idea–it’s a similar business model to HBO. But Netflix seems to have the execution right, at least with this first foray.

Perhaps the biggest difference with convention television is that it doesn’t matter how many people watched House of Cards during its debut week. As Hastings said in a letter to investors two weeks ago:

Linear channels must aggregate a large audience at a given time of day and hope the show programmed will actually attract enough viewers despite this constraint. With Netflix, members can enjoy a show anytime, and over time, we can effectively put the right show in front of members based on their viewing habits. Thus we can spend less on marketing while generating higher viewership.

For linear TV, the fixed number of prime-time slots mean that only shows that hit it big and fast survive, thus requiring an extensive and expensive pilot system to keep on deck potential replacement shows. In contrast, Internet TV is an environment where smaller or quirkier shows can prosper because they can find a big enough audience over time. In baseball terms, linear TV only scores with home runs. We score with home runs too, but also with singles, doubles and triples.

Because of our unique strengths, we can commit to producing and publishing “books” rather than “chapters”, so the creators can concentrate on multi-episode story arcs, rather than pilots. Creators can work on episode 11 confident that viewers have recently enjoyed episodes 1 to 10. Creators can develop episodes that are not all exactly 22 or 44 minutes in length. The constraints of the linear TV grid will fall, one by one.

I look forward to seeing more of this strategy, and as I proceed with House of Cards you may even get a post on its politics.

Was the Civil War a Constitutional Fork?

Confederate ConstitutionShortly after Aaron Swartz’s untimely suicide, O’Reilly posted their book Open Government for free on Github as a tribute. The book covers a number of topics from civil liberties and privacy on the web to how technology can improve government,  with each chapter written by a different author. My favorite was the fifth chapter by Howard Dierking. From the intro:

In many ways, the framers of the Constitution were like the software designers of today. Modern software design deals with the complexities of creating systems composed of innumerable components that must be stable, reliable, efficient, and adaptable over time. A language has emerged over the past several years to capture and describe both practices to follow and practices to avoid when designing software. These are known as patterns and antipatterns.

The chapter goes on to discuss the Constitution and the Articles of Confederation as pattern and antipattern, respectively. In the author’s own words he hopes to “encourage further application of software design principles as a metaphor for describing and modeling the complex dynamics of government in the future.”

In the spirit of Dierking’s effort, I will offer an analogy of my own: civil war as fork. In open source software a “fork” occurs when a subset of individuals involved with the project take an existing copy of the code in a new direction. Their contributions are not combined into the main version of the project, but instead to their new code base which develops independently.

This comparison seems to hold for the US Civil War. According to Wikipedia,

In regard to most articles of the Constitution, the document is a word-for-word duplicate of the United States Constitution. However, there are crucial differences between the two documents, in tone and legal content, and having to do with the topics of states’ rights and slavery.

Sounds like a fork to me. There’s a full list of the “diffs” (changes from one body of text or code to another) on the same wiki page. But to see for myself, I also put the text of the US Constitution on Github, then changed the file to the text of the CSA Constitution. Here’s what it looks like visually:


As the top of the image says, there are 130 additions and 119 deletions required to change the US Constitution into that of the Confederacy. Many of these are double-counts since, as you can see, replacing “United States” with “Confederate States” counts as both a deletion of one line and an addition of a new one.

I did not change trivial differences like punctuation or capitalization, nor did I follow the secessionists’ bright idea to number all subsections (which would have overstated the diffs). Wikipedia was correct that most of the differences involve slavery and states’ rights. Another important difference is that the text of the Bill of Rights is included–verbatim–as Section 9 of Article 1 rather than as amendments.

In other words, the constitution of the CSA was a blatant fork of the earlier US version. Are there other cases like this?

What Did Manifest Destiny Look Like?

“Manifest Destiny was the belief widely held by Americans in the 19th century that the United States was destined to expand across the continent. The concept, born out of ‘a sense of mission to redeem the Old World’, was enabled by ‘the potentialities of a new earth for building a new heaven.'” (Wikipedia, citing Frederick Merk)

Now, Michael Porath has told the story of manifest destiny in a series of 141 maps. The main technical trick is that Porath designed the site in HTML5, so it has some nice interactive features. The maps appear on a single page in four columns but you can click any of them for a close-up with an explanation of the changes, or mouse-over a region of the map to see what political entity it was under at the time (e.g. unorganized territory, Spanish colony).

There are two additions that I think would help improve this project. The first is a sense of time scale–some of the maps are only a month apart (January and February 1861, for example) while others are separated by several decades (March 1921 and January 1959). Adding time would allow for a second feature: an animation that would show the areas of change and continuity over time. An excellent example of this is David Sparks’ choropleth maps of presidential voting over time.  I do not know whether this could still be done in Porath’s HTML5 setup, but it is often useful to think about changes to graphical displays (additions or subtractions) that would help to convey meaningful information. What other suggestions do you have for these maps?

What Are the Chances Your Vote Will Matter?

Only one vote matters. In the United States, the vote that gives a presidential candidate the majority in the state that tips the electoral college decides it all. Nevertheless, about 122 million US voters went to the polls for the 2008 Presidential election.

If the only benefit you get from voting is your candidate winning, this behavior is totally irrational. Voters spend precious time and effort traveling to the polls or arranging for mail-in ballots, with very small odds that this will make any difference in the final outcome. Of course, the simplest explanation is that this argument is wrong and voting can be rational, but you could also say that voting is self-expression.

In a recent paper (gated), Douglas VanDerwerken
takes a slightly different approach. He estimates a one in 2.5 million chance that his vote will matter this year, given that he lives in North Carolina (a competitive state in 2008, and likely in 2012 too).* But then he points out that, “Even if your vote does not have an effect on the election, it can certainly have an effect on you.” His broader message is that:

Statistics is not divorced from subjectivity, nor from morality. What you decide depends on your moral axioms.

We can use statistics to inform our objective calculations, and our subjective intuitions, but decision-making is not a “plug and chug” process. In summarizing data, the statistician makes important decisions about how to abstract away from reality and what message to send. When that information as inputs for further decision-making–which always involves trade-offs–the statistician bears some responsibility for the outcome. Once again we are reminded that statistics is a rhetorical practice. (See also here and here.)


*Full disclosure: Doug teaches the lab section of a Duke statistics course in which I am currently enrolled.

PolMeth 2012 Round-Up, Part 2

A Map from Drew Linzer’s Votamatic

Yesterday I discussed Thursday’s papers and posters from the 2012 Meeting of the Political Methodology Society. Today I’ll describe the projects I saw on Friday, again in the order listed in the program. Any attendees who chose a different set of panels are welcome to write a guest post or leave a comment.

First I attended the panel for Jacob Montgomery and Josh Cutler‘s paper, “Computerized Adaptive Testing for Public Opinion Research.” (pdf; full disclosure: Josh is a coauthor of mine on other projects, and Jacob graduated from Duke shortly before I arrived) The paper applies a strategy from educational testing to survey research. On the GRE if you get a math problem correct, the next question will be more difficult. Similarly, when testing for a latent trait like political sophistication a respondent who can identify John Roberts likely also recognizes Joe Biden. Leveraging this technique can greatly reduce the number of survey questions required to accurately place a respondent on a latent dimension, which in turn can reduce non-response rates and/or survey costs.

Friday’s second paper was also related to survey research: “Validation: What Big Data Reveal About Survey Misreporting and the Real Electorate” by Stephen Ansolabehere and Eitan Hersh (pdf). This was the first panel I attended that provoked a strong critical reaction from the audience. There were two major issues with the paper. First, the authors contracted out the key stage in their work–validating data by cross-referencing other data sets–to a private, partisan company (Catalist) in a “black box” way, meaning they could not explain much about Catalist’s methodology. At a meeting of methodologists this is very disappointing, as Sunshine Hillygus pointed out. Second, their strategy for “validating the validator” involved purchasing a $10 data set from the state of Florida, deleting a couple of columns, and seeing whether Catalist could fill those columns back in. Presumably they paid Catalist more than $10 to do this, so I don’t see why that would be difficult at all. Discussant Wendy Tam Cho was my favorite for the day, as she managed to deliver a strong critique while maintaining a very pleasant demeanor.

In the afternoon, Drew Linzer presented on “Dynamic Bayesian Forecasting of Presidential Elections in the States” (pdf). I have not read this paper, but thoroughly enjoyed Linzer’s steady, confident presentation style. The paper is also accompanied by a neat election forecast site, which is the source of the graphic above. As of yesterday morning, the site predicted 334 electoral votes for Obama and 204 for Romney. One of the great things about this type of work is that it is completely falsifiable: come November, forecasters will be right or wrong. Jamie Monogan served as the discussant, and helped to keep the mood light for the most part.

Jerry Reiter of the Duke Statistics Department closed out the afternoon with a presentation on “The Multiple Adaptations of Multiple Imputation.” I was unaware that multiple imputation was still considered an open problem, but this presentation and a poster by Ben Goodrich and Jonathan Kropko (“Assessing the Accuracy of Multiple Imputation Techniques for Categorical Variables with Missing Data”) showed me how wrong I was. Overall it was a great conference and I am grateful to all the presenters and discussants for their participation.

PolMeth 2012 Round-Up, Part 1

Peter Mucha’s Rendering of Wayne Zachary’s Karate Club Example

Duke and UNC jointly hosted the 2012 Meeting of the Society for Political Methodology (“PolMeth”) this past weekend. I had the pleasure of attending, and it ranked highly among my limited conference experiences. Below I present the papers and posters that were interesting to me, in the order that I saw/heard them. A full program of the meeting can be found here.

First up was Scott de Marchi‘s piece on “Statistical Tests of Bargaining Models.” (Full disclosure: Scott and most of his coauthors are good friends of mine.) Unfortunately there’s no online version of the paper at the moment, but the gist of it is that calculating minimum integer weights (MIW) for the bargaining power of parties in coalition governments has been done poorly in the past. The paper uses a nice combination of computational, formal, and statistical methods to substantially improve on previous bargaining models.

Next I saw a presentation by Jake Bowers and Mark Fredrickson on their paper (with Costas Panagopoulos) entitled “Interference is Interesting: Statistical Inference for Interference in Social Net- work Experiments” (pdf). The novelty of this project–at least to me–was viewing a treatment as a vector. For example, given units of interest (a,b,c), the treatment vector (1,0,1) might have different effects on a than (1,1,0) due to network effects. In real-world terms, this could be a confounder for an information campaign when treated individuals tell their control group neighbors about what they heard, biasing the results.

The third paper presentation I attended was “An Alternative Solution to the Heckman Selection Problem: Selection Bias as Functional Form Misspecification” by Curtis Signorino and Brenton Kenkel. This paper presents a neat estimation strategy when only one stage of data has been/can be collected for a two-stage decision process. The downside is that estimating parameters for a k-order Taylor series expansion with n variables grows combinatorically, so a lot of observations are necessary.* Arthur Spirling, the discussant for this panel, was my favorite discussant of the day for his helpful critique of the framing of the paper.

Thursday’s plenary session was a talk by Peter Mucha of the UNC Math Department on “Community Detection in Multislice Networks.” This paper introduced me to the karate club example, the voter model, and some cool graphs (see above).

At the evening poster session, my favorite was Jeffrey Arnold‘s  “Pricing the Costly Lottery: Financial Market Reactions to Battlefield Events in the American Civil War.” The project compares the price of gold in Confederate graybacks and Union greenbacks throughout the Civil War as they track battlefield events. As you can probably guess, the paper has come cool data. My other favorite was Scott Abramson‘s labor intensive maps for his project “Production, Predation and the European State 1152–1789.”

I’ll discuss the posters and papers from Friday in tomorrow’s post.


*Curtis Signorino sends along a response, which I have abridged slightly here:

Although the variables (and parameters) grow combinatorically, the method we use is actually designed for problems where you have more regressors/parameters than observations in the data.  That’s obviously a non-starter with traditional regression techniques.  The underlying variable selection techniques we use (adaptive lasso and SCAD) were first applied to things like trying to find which of thousands of genetic markers might be related to breast cancer.  You might only have 300 or a 1000 women in the data, but 2000-3000 genetic markers (which serve as regressors).  The technique can find the one or two genetic markers associated with cancer onset.  We use it to pick out the polynomial terms that best approximate the unknown functional relationship.  Now, it likely won’t work well with N=50 and thousands of polynomial terms.  However, it tends to work just fine with the typical numbers of regressors in poli sci articles and as little as 500-1000 observations.  The memory problem I mentioned during the discussion actually occurred when we were running it on an IR dataset with something like 400,000 observations.  The expanded set of polynomials required a huge amount of memory.  So, it was more a memory storage issue due to having too many observations.  But that will become a non-issue as memory gets cheaper, which it always does.

This is a helpful correction, and perhaps I should have pointed out that there was a fairly thorough discussion of this point during the panel. IR datasets are indeed growing rapidly, and this method helps avoid an almost infinite iteration of “well, what about the previous stage…?” questions that reviewers could pose.

Petition for TSA to Obey the Law

Thousands Standing Around in Denver

From Jim Harper (via Josh Cutler):

A year ago this coming Sunday, the US Court of Appeals for the DC Circuit ordered the Transportation Security Administration to do a notice-and-comment rulemaking on its use of Advanced Imaging Technology (aka “body-scanners” or “strip-search machines”) for primary screening at airports. (The alternative for those who refuse such treatment: a prison-style pat-down.) It was a very important ruling, for reasons I discussed in a post back then. The TSA was supposed to publish its policy in the Federal Register, take comments from the public, and issue a final ruling that responds to public input.

And the wording of the petition:

In July 2011, a federal appeals court ruled that the Transportation Security Administration had to conduct a notice-and-comment rulemaking on its policy of using “Advanced Imaging Technology” for primary screening at airports. TSA was supposed to publish the policy in the Federal Register, take comments from the public, and justify its policy based on public input. The court told TSA to do all this “promptly.” A year later, TSA has not even started that public process. Defying the court, the TSA has not satisfied public concerns about privacy, about costs and delays, security weaknesses, and the potential health effects of these machines. If the government is going to “body-scan” Americans at U.S. airports, President Obama should force the TSA to begin the public process the court ordered.

You can sign the petition, or read more about the ineffectiveness of current TSA procedures here.

Wednesday Nerd Fun: The Sounds of America

DARE’s Linguistic Map of the US

The Dictionary of American Regional English (DARE) is a project initiated almost 50 years ago to document “words, phrases, and pronunciations that vary from one place to another place across the United States.” The map at the right gives a sense of how much variation field interviewers found between 1965 and 1970. Beginning with those interviews DARE has grown to five volumes, the last of which is now available.

One technique that the interviewers used to record regional dialects was a story called “Arthur the Rat.” The story’s main purpose was to include almost all of the sounds of American English when read aloud. A sample recording includes speakers from Brooklyn, Boston, Memphis, and rural areas across the country. Over 800 recordings were made, all of which have been digitized in a collection at the University of Wisonsin.

The DARE website also includes features such as quizzes to test your knowledge of American English and a word of the month. Did you know google dates back to 1859? Happy 4th of July!