Dropsonde: predicting the world

Dropsonde was my first real engineering project when I was just starting out thinking about how to built interesting, intelligent systems. I think I was working on it circa 2010. A lot of my thinking and worldview has changed since then.

Dropsonde came about after re-reading William Gibson's Idoru; about the same time, I read an article that described a situation in which a bomb threat in downtown London sent waves through Twitter 45 minutes before the story was picked up by news outlets. This was remarkable to me, and I sought to build a system that might pick up these tremors and perhaps allow me to identify major events before they occurred. This is, of course, a grandiose vision; perhaps all undertakings worth doing start out similarly. Prior to this, I had only written trivial things in Perl and a few systems daemons in C. I had, however, just begun learning Python for a job I had just started.

It was Python that really allowed me to explore the field of software engineering. It has a very simple syntax and a very rich ecosystem, allowing me to switch between assembling systems from blocks (to focus on the behaviour of the system as a whole) and building components (to focus on implementing the pieces). I started by sketching my idea on paper, determining what it was I was to build. Prior to this, I had never really interacted with a database, built so large a system (comparatively), or heard of natural language processing; yet, in the course of building dropsonde, I would delve into each of these. It served as a foundation from which to explore other ideas, and a precedent for learning how to understand what I didn't know.

Why dropsonde?

The reasoning behind dropsonde is difficult to elucidate, and its aims even more difficult to articulate. There are two quotes by William Gibson in his book Idoru that really capture some of the essence of what I wanted to accomplish:

… he wondered vaguely if there might be a larger system, a field of greater perspective. Perhaps the whole of DatAmerica possessed its own nodal points, info-faults that might be followed down to some other kind of truth, another mode of knowing, deep within gray shoals of information. But only if there were someone there to pose the right question.

and

Slitscan itself, Laney suspected, might be one of those larger nodal points he sometimes found himself trying to imagine, an informational peculiarity opening into some unthinkably deeper structure.

The basic tenet behind dropsonde is that all of reality is the structure and interpretation of data. As any data researcher will tell you, this is not a clinical fact but one that yields much beauty. The interpretation of these data structures is what yields meaning to reality. The aim of dropsonde, therefore, is to identify socio-informatic undercurrents, alternative angles from which to discover new structures in the data. If these undercurrents can be identified, it may be possible for hackers to exist outside the context of society (a concept not yet able to be articulated, although work on this is underway). Some of the ensuing unanswered questions:

What kinds of alternative data structures might exist? What would they look like? How will they be identified?
What are some of the current data structures?
What is the context of society?
What are the implications of existing outside this context?
What can we do with this new mode of knowing?

The name dropsonde refers to the sensor packages dropped into storms to more accurately understand the causes of the storm. The dropsonde software deploys sensors into the data stream to more accurately study social patterns and the triggers for trends, the so-called nodal points. At this point, the exact usefulness of this new perspective is unknown, but it is believed that it is worth pursuing. Call it a hunch.

This leads directly into the next question.

What is dropsonde?

At its core, dropsonde is a datamining backend paired with tools for data analysis. Identifying nodal points is done in conjunction with a human analyst: whereas the ability to identify these nodal points is most likely a human skill, the software may be able to identify candidates, sifting through an amount of data that would overwhelm the human analyst.

The first version used "crawlers"–a daemon connected to multiple twitter accounts, each targeted to one of several subject areas:

politics and business: this is a fine line for a union, but in the future, these subjects may be split out to separate accounts.
technology and science
art and fashion: a useful predictor (and indicator) of culture makers and hackers
public feed (taps into the at large data stream for overall trending)

These crawlers served as the sensors, as the dropsondes proper.

The daemon stored status updates and user information to a postgres database; a report generator ran on this every morning and extracted some basic statistical information, such as:

trending stocks (interestingly enough, playing with a small portfolio of dropsonde-picked stocks indicated small-but-appreciable growth over a three-month period)
most mentioned links
most mentioned users

Try as I might, I can't find any of the old reports.

The daemon also employed primitive network expansion, attempting to balance the quantity of data sources (aka twitter users) with quality data sources (users with some reputation or influence behind them). This was mainly done by cross referencing social networks of other users: for example, identifying users common in the network but not currently being followed by a dropsonde crawler.

dropsonde mkII aimed to expand the data sources from the original dropsonde to include not only twitter, but RSS/atom feeds as well. Once an interface had been established for getting data into the system, other sources would be added. dropsonde mkII was started during my time in Africa, but never got anywhere.

Summary

#+fbegin_quote All of reality distills down to the structure and interpretation of data; the world is but a series of data structures whose interpretation lends meaning to existence. Dropsonde aims to investigate possible unseen angles - new interpretations - and alternative structures for information in order to attain some sort of greater knowing or understanding of the world. Perhaps in these data structures, it would be possible for cyberpunks to exist outside the context of society, if that were something to be interested in. #+end_quote
Need to clarify
- alternative sociopolitical data structures
- the context of society
- the implications of existing outside that context

The original Dropsonde notes

These are the original notes that started the project, scribbled in Xournal on my n810 while on the bus and during a conversation with Matt Bailey and Aaron Bieber.

2010-10-15

Social Intelligence

predict / identify / analyze critical events
identify key nodes
map networks / ∂t
guerilla ISR in information networks
detection & analysis of tremors even in public realm can provide useful intelligence generators

2010-10-16

Goals of Dropsonde:

identify key nodes
- events
- agents
- trends
- social networks * info flow * network connectivity * perturbations (butterfly effect)
predict changes in nodality

Data modeling questions
- how to represent nodes
- how can nodes and networks be usefully and accurately visualized?
Miscellanea:
- agent based modeling
- swarm.org
- archaeoinformatics.org
Notes:
1. ISR: Intelligence / surveillance / reconnaissance. This is a term from the military that carries a specific meaning. It includes things like drone reconnaissance, sniper teams and listening/observation posts, PSYOP teams to assess local populace' psychological state, etc….