Policy Algorithms

Text Mining and Policy

Text Mining has come a long way. The true test for me is to use it to uncover the effectiveness of policies.

Science fiction author Isaac Asimov was the first to introduce me to text analysis. In his Foundation Trilogy scientists and engineers preserve their knowledge in view of the fall of the galactic empire. In one scene, an imperial diplomat visits the leader of the Foundation. After he leaves, scientists analyze what their visitor has said and which terms they negotiated. They conclude that he has said nothing and promised nothing after three days of talks.

Text mining has automated the process of analyzing documents. Examples you may know are sentiment analysis of social media and product reviews. Before this process was digitized, an impressive feat was accomplished: every word in the English language was scored on the distance of their meaning to other words. So, good and bad have the maximum distance score. That way, positive words are nearer ‘good’ than ‘bad’ and vice versa.

This TED video ( shows the evolution of words in books.

Hopefully, text mining policy documents would yield information on policies’ effectiveness and the conditions for success. We’ve moved a long way from empty galactic talks!