Here’s a great piece from PBS.

A new set of tools would help reporters find patterns in otherwise unstructured or unsearchable information. For instance, the Obama administration posted letters from dozens of interest groups providing advice on issues, but the letters were not searchable. A text-extraction tool would allow reporters to feed PDF documents into a Web service and return a version that could be indexed and searched. The software might also make it easy to tag documents with metadata such as people’s names, places and dates. Another idea is to improve automatic transcription software for audio and video files, often available (but not transcribed) for government meetings and many court hearings.

Wired UK goes a bit deeper into some specific companies and projects.

And here’s a nice presentation by Kurt Cagle that gives a good overview of some of the computational foundational technology out there.

I don’t think it’s unreasonable to think that the vast majority of daily news will be completely machine generated and machine broadcast.  Journalists will be increasingly involved in bigger, deeper features and defining the computational logic to generate the news stream.

Wolfram|Alpha iPhone App

[full disclosure: I've been working with the Wolfram|Alpha team... so I'll leave out a review or discussion on price justification]

The Wolfram|Alpha iPhone App is live in the App Store.  The blogosphere has its own impressions.  Argue price, features, business models, and whatever else that seems relevant in those lively communities.  On this blog and post I’ll give you my impressions and use cases.

Features that are unique and handy to the iPhone App:

• History – it keeps a running history of your queries.  As a utility for doing many queries within the same domain, this is extremely helpful and time saving
• Favorites – As with the history, it’s nice to set up your own list of queries you’ll do over and over again.  This becomes a sort of “homepage” for me.
• better GeoLocation – because the Apps can get at lat/long the overall geolocating is stronger than via the geoIP on the website.
• Refined Keyboard input – when doing a lot of math or long calculations it’s much easier to input queries with a fine-tuned keyboard.  Though not a limitation on the full website, the constraints of the default mobile keyboards made using the website challenging on a smart phone.  Not so in the app.
• Direct Links to Source Information – this will go unnoticed by many… but having a hyper linked bibliography with this depth is extremely useful if you need to research deeply or go beyond the calculation.

Features that make this a killer mobile app for me:

• Instant access to real time information about the most important daily data points in my life all in one app: weather, stocks, earthquakes (i live in la!) …
• An easy to use calculator that saves me tons of keystrokes via unit conversions and cleaning up my sloppy input
• deep data in many of my own hobbies, pursuits and professional domains.  Getting an individual app for each hobby is quite expensive, time consuming and frustrating to learn how apps work

Things that can be improved:

• I wish I could set the Favorites tab to be my default opening screen with a search box on top.  This would speed up activation of the service
• Offline use for very basic calculations when no compute data is accessed.
• Ability to Add Notes – I’d like to be able to keep private comments and ideas right inside the app (and export later with links to the inputs/outputs)
• Mashing with Wikipedia and/or other essay style knowledge sources – would be convenient to have backgrounders on some topics and be able to use the Wolfram|Alpha assumption/categorization to do some of the knowledge mining for me.

Post your impressions/use cases once you get the app.

It could be we see what words we have in english with OL as the leading letters, assuming the rest of the google letters will be used.  Here’s what Wolfram|Alpha has…

Maybe we should remove those letters from Google, and see what we can make.  G_OG_E??? what else has those letters?

By the looks of this I don’t think it’s going to be some obvious word or just “google” letters.

What’s fun is that you can actually ATTEMPT to run these computations in Wolfram|Alpha.  Here’s mercury in 1 billion years. Unfortunately the one thing I want to be able to show is the orbits of the planets and that is pushing W|A to the heuristic timing limit.

I can put this into Mathematica and work it out using more local CPU power.  Then again, I like just playing with numbers to see where I can take this.  Here’s Mercury at 199,999 years.  Things get gnarly.

Check this graph out… think there’s a relationship?

GM revenue vs US Carbon Emissions

cool. very cool.

Asking which result is “right” misses the point. Google is a search engine; it did exactly what it’s supposed to do. It isn’t making any any assumptions about what you’re looking for, and will give you everything the cat dragged in. If you’re an elementary school teacher or a flat-earther, you can find the result you want somewhere in the big, messy pile. If you want accurate data from a known and reliable source, and you want to use that data in other computations, you don’t want Google’s answer; you want Alpha’s. (BTW, the Earth’s circumference is .1024 of the distance to the Moon.)

When is this important? Imagine we were asking a more politically charged question, like the correlation between childhood vaccinations and autism, or the number of civilians killed in the six-day war. Google will (and should) give you a wide range of answers, from every part of the spectrum. It’s up to you to figure out whether the data actually came from. Alpha doesn’t yet have data about autism or six-day war casualties, and even when it does, no one should blindly assume that all data that’s “curated” is valid; but Wolfram does its homework, and when data like this is available, it will provide the source. Without knowing the source, you can’t even ask the question.

But what about all the actual knowledge that we as humans have accumulated?

A lot of it is now on the web—in billions of pages of text. And with search engines, we can very efficiently search for specific terms and phrases in that text.

But we can’t compute from that. And in effect, we can only answer questions that have been literally asked before. We can look things up, but we can’t figure anything new out.

Let’s see where this goes!

Wolfram Mathematica Home Edition is available.  It’s a $295 fully functional version of Mathematica 7.

Everyone should consider getting a copy.  No, really, everyone.  

What mathematica can help you do is as useful as word processing.  I know, that sounds crazy.  How could scientific computing be for everyone?

Consider the amount of math, data mining and research one already does just to get through the day.  Do you check the stock market? do you look up information in wikipedia? do you use the tools in your online bank site? Do you watch the weather report?

Much of this data is available in Mathematica and is immediately made interactive by Mathematica.  Other examples

• Map the planets and dwarf planets of our solar system, along with stars, galaxies, and more
• Calculate expected returns for a stock or examinethe stock market’s performance over time
• Manipulate images in sophisticated ways
• Make science and math, from preschool through college and beyond, come alive
• Visualize worldwide weather patterns or see decades of tornado data
• Examine protein alignments or algae growth
• Play with map projections or record your travels
• Pursue your interests in number theory or visualizecomplex functions
• Design buildings and create art
• Decorate Easter eggs or draw a Valentine
• And much, much more

OK, still not convinced?  Just do the math.  Mathematica can replace Visio, your calculator (graphing calculator), excel, batch photo editor and most common programming environments.

If you a developer, even just a dabbler, you must get Mathematica.  It’s easy to pick up and the more you learn the more amazing things you find.  Beyond that though, Mathematica’s symbolic programming is a progressive approach.  In a world of multi core, multi threaded apps OOP and Procedural programming is becoming increasingly complicated and bug prone.  Mathematica’s approach avoids the pitfalls of lost threads and memory leaks because the paradigm itself doesn’t allow you to make those mistakes (for the most part).  

I’ll let you in on another secret, that almost no literature covers.  Mathematica has the best web parsers out there.  It is insanely easy to bring data in from like 200 different file formats, including HTML.  For anyone who has ever built a web service, a scraper, spider or crawler, you know how painful it is to build these in most languages, not to mention maintaining a scraper or crawler.  Why no one promotes this feature is beyond me considering the mashup nature of the Web now.  It’s super fun to mash the various APIs out there with some cool mathematica visualizations.  (Oh, and for the search engine nutz out there, the linguistic engine in mathematica is insanely easy to use vs. raw wordnet and various spelling engines.  you can creating a really neat search suggestion tool within in an hour.)

(e.g. I made a visual search engine of shoes and women’s tops that crushes like.com.  it took me 1.5 hours.  I used the image manipulation tools in Mathematica to analyze shapes and colors of products via the built in similarity algorithms.  Post a comment if you want that code)

So, yes, web industry people/media workers, you can get way ahead with this software.

BI people.  Give up that lame copy of SAS and SPSS.  Seriously, those products are so expensive for somewhat limited use.  I’ll still install R, because it’s FREE and extensible, but those other two gotta go if you are a stats and BI person.  Get a home copy of mathematica, learn it, and then get a pro copy at work.  Don’t trust me on this, just try it.  Let me know if you really can’t kick your SPSS habit.

I really could go on forever.  The scope of use for this software is pretty insane.  Hell, the documentation alone is a great teaching aid.  Sometimes I just browse the documentation to learn new math or programming or to explore the data.  What few people know is that the documentation itself is interactive and computable.  You don’t just get a book of examples, you can actually “run the program” within the documentation and see it live.  For the home user, this means you can use the documentation to get going very quickly and start to modify the examples to suit your task.

Call me a FanBoy.  That’s fine.  You will be too if you invest $295 and 2 hours of your time.  Methinks you’ll feel what I feel about this – how can I possibly be given this much power without paying 10x this much?  There must be a catch!  There isn’t.  This is the best deal in software. (just think of how much you paid for MS Office and Photoshop… and those only do a handful of functions)

I’m in agreement with this:

At some point you have to go look at the actual world. And when you do, you will find yourself at the bottom of a well looking out through a straw at an immensity. Honesty and rationality start with an elementary humility in the face of this inescapable fact. This does not mean you can’t know things; you can. It does mean you can’t know everything.

It’s a challenging post which befits the challenge of uncovering the limits of knowledge, quantum computer and “just how does this universe thing work.”

We can’t know everything, but there’s no limit to how much we can know (meaning, there’s not enough time in our lifetimes for us to reach even the formal limit!).  Chew on that.

Sure, I’ll have far more details in upcoming posts, though most of those details will involve actual math, code, projects and implications and less about the school experience.

~Russ