Feeds:
Posts
Comments

Posts Tagged ‘programming’

We have a problem.

As it stands now the present and near future of economic, social and cultural development primarily derives from computers and programming.   The algorithms already dominate our society – they run our politics, they run our financial system, they run our education, they run our entertainment, they run our healthcare.    The ubiquitous tracking of everything that can possible be tracked determined this current situation.   We must have programs to make things, to sell things, to exchange things.

punchcard

The problem is not necessarily the algorithms or the computers themselves but the fact that so few people can program.    And why?   Programming Sucks.

Oh sure, for those that do program and enjoy it, it doesn’t suck. As Much.   But for the 99%+ of the world’s population that doesn’t program a computer to earn a living it’s a terrible endeavour.

Programming involves a complete abstraction away from the world and all surroundings.  Programming is disembodied – it is mostly a thought exercise mixed with some of the worst aspects of engineering.   Mathematics, especially the higher order really crazy stuff was long ago unapproachable and completely disembodied requiring no physical engineering or representation at all.  Programming, in most of its modern instances, consequences very far away from its creative behavior.  That is, in most modern system it takes days, weeks, months years to personally feel the results deeply of what you’ve built.    Programming is ruthless.  It’s unpredictable.   It’s 95% or more reinventing the wheel and configuring environments to even run the most basic program.  It’s all set up, not a lot of creation.   So few others understand it they can’t appreciate the craft during the act (only the output is appreciated counted in users and downloads).

There are a couple of reasons why this is the case – a few theoretical/natural limits and a few self-imposed, engineering and cultural issues.

First the engineering and cultural issues.   Programming languages and computers evolved rather clumsily built mostly by programmers for other programmers – not for the majority of humans.    There’s never been a requirement to make programming itself more humanistic, more embodied.    Looking back on the history of computers computing was done always in support of something else, not for its own sake.   It was done to Solve Problems.   As long as the computing device and program solved the problem the objective was met.   Even the early computer companies famously thought it was silly to think everyone one day might actually use a personal computer.   And now we’re at a potentially more devastating standstill – it’s absurd to most people to think everyone might actually need to program.    I’ll return to these issues.

Second the natural limits of computation make for a very severe situation.   There are simply things that are non-computable.   That is, we can’t solve them.   Sometimes we can PROVE we can’t solve them but that doesn’t get us any closer to solving some things.    This is sometimes called the Halting Problem.  The idea is basically that for a sufficiently complex program you can’t predict whether the program will halt or not.   The implication is simply you must run the program and see if it halts.  Again, complexity is the key here.  If these are relatively small, fast programs with a less than infinite number of possible outcomes then you can simply run the program across all possible inputs and outputs.   Problem is… very few programs are that simple and certainly not any of the ones that recommend products to you, trade your money on wall street, or help doctors figure out what’s going on in your body.

STOP.

This is a VERY BIG DEAL.    Think about it.   We deploy millions of programs a day with completely non-deterministic, unpredictable outcomes.  Sure we do lots of quality assurance and we test everything we can and we simulate and we have lots of mathematics and experience that helps us grow confident… but when you get down to it, we simply don’t know if any given complex program has some horrible bug in it.

This issue rears its head an infinite number of times a day.   If you’ve ever been mad at MS Word for screwing up your bullet points or your browser stops rendering a page or your internet doesn’t work or your computer freezes… this is what’s going on.  All of these things are complex programs interacting with other programs and all of them have millions (give or take millions) of bugs in them.  Add to it that all of these things are mutable bits on your computer that viruses or hardware issues can manipulate (you can’t be sure the program you bought is the program you currently run) and you can see how things quickly escape our abilities to control.

This is devastating for the exercise of programming.  Computer scientists have invented a myriad of ways to temper the reality of the halting problem.   Most of these management techniques makes programming even more mysteries and challenging due to the imposition of even more rules that must be learned and maintained.   Unlike music and writing and art and furniture making and fashion we EXPECT and NEED computers to do exactly what we program them to do.   Most of the other stuff humans do and create is just fine if it sort of works.  It still has value.  Programs that are too erratic or worse, catastrophic, are not only not valuable we want to eliminate them from the earth.   We probably destroy some 95%+ of the programs we write.

The craft of programming is at odds with its natural limits.   Our expectations and thus the tools we craft to perform program conflict with the actuality.  Our use of programs exceeds their possibilities.

And this really isn’t due to computers or programming, but something more fundamental: complexity and prediction.    Even as our science shows us more and more that prediction is an illusion our demands of technology and business and media run counter.    This fundamental clash manifests itself in programming, programming languages, the hardware of computers, the culture of programming.  It is at odds with itself and in being so conflicted is unapproachable to those that don’t have ability to stare maddeningly into a screen flickering with millions of unknown rules and bugs.   Mastery is barely achievable except for a rare few.   And without mastery enjoyment rarely comes – the sort of enjoyment that can sustain someones attention long enough to do something significant.

I’ve thought long and hard about how to improve the craft of programming.   I’ve programmed a lot, lead a lot of programming efforts, delivered a lot of software, scrapped a lot more.  I’ve worked in 10+ languages.  I’ve studied mathematics and logic and computer science and philosophy.  I’ve worked with the greatest computer scientists.  I’ve worked with amazing business people and artists and mathematicians.   I’ve built systems large and small in many different categories.  In short, I’ve yet to find a situation in which programming wasn’t a major barrier to progress and thinking.

The solution isn’t in programming languages and in our computers.  It’s not about Code.org and trying to get more kids into our existing paradigm. This isn’t an awareness or interest problem.   The solution involves our goals and expectations.

We must stop trying to solve every problem perfectly.  We must stop trying to predict everything.   We must stop pursuing The Answer, as if it actually exists.  We must stop trying to optimize everything for speed and precision and accuracy. And we must stop applying computerized techniques to every single category of activity – at least in a way where we expect the computer to forever to the work.

We must create art.  Programming is art.  It is full of accidents and confusions and inconsistencies.   We must turn it back to an analog experience rather than a conflicted digital.    Use programming to explore and narrate and experiment rather than answer and define and calculate.

The tools that spring from those objectives will be more human.  More people will be able to participate.  We will make more approachable programs and languages and businesses.

In the end our problem with programming is one of relation – we’re either relating more or less to the world around us and as computers grow in numbers and integration we need to be able to commune, shape and relate to them.

Read Full Post »

We all are programmers.   And I want to explain what programming really is.  Most people think of it as a writing instructions that a computer will then interpret and go do what those instructions say.   In only the simplest sense is this fully encompassing of what programming is.

 

Programming in the broadest sense is a search through computational universe for interesting patterns that can be interpreted by other patterns.   A few definitions are in order.   A pattern is simply some set of data pulled from the computational universe (from my own investigations/research/logic everything is computational).  Thus a pattern could be a sentence of English words or a fragment of a program written in Java or DNA strands or a painting or anything else.   Some patterns are able to interact with other patterns (information processing) such as a laptop computer can interpret Microsoft Office documents or a replicated set of DNA (a human) can interpret Shakespeare and put on a play.   A program is simply a pattern that interacts with other programs.

 

When we write programs we are simply searching through the space of symbolic representations in whatever programming language.   When a program doesn’t work/doesn’t do what we want, we haven’t found a pattern of symbols that’s interpreted the way we prefer or the processing pattern can interpret.  We sometimes call that “bugs” in the software.   Underneath it all it’s simply another program, just not the one we want.

 

I call it a search to bring particular activities to mind.  When we say we write a program or create a program it seems to engender only a limited set of methods to find programs by a limited set of people, called programmers.   Calling it a search reflects reality AND opens our eyes to the infinite number of ways to find interesting patterns and to interpret them.   The space of programs is “out there”, we just have to mine it for the programs/patterns we wish to interpret.

 

Programs/patterns that become widely used owe that use to the frequency that those patterns can be interpreted.  For example, Windows or MacOS have billions of interpreting machines in which their programs can be interpreted.   Or on an even bigger scale, DNA “programs” have trillions of interpreters on just this planet alone.

 

Using a program is nothing more than interpreting it.  When you type a document in MS Word the OS is interpreting your keystrokes, refreshing the screen with pixels that represent your words, all while MS word itself is checking for grammar put in place by programmers who interpreted a grammar reference and so on and so on.   For sufficiently complex programs we aren’t able to say if a program “does the right thing.”.  Only simple programs are completely verifiable.   This is why programs exist only as patterns that are interpreted.

 

Humans have become adept at interpreting patterns most useful for the survival of human genes.  With the advent of digital computers and related patterns (tech) we are now able to go beyond the basic survival of our genes and instead mine for other patterns that are “interesting” and interpretable by all sorts of interpreters.  I don’t know where the line is on good for survival and not, but it’s really not a useful point here.  My point is that with computers we’re able to just let machines go mining the space of existence in much grander ways and interpreting those results.   Obvious examples include the SETI project mining for signs of aliens, LHC mining the space of particle collisions, Google search mining the space of webpages and now human roadways, Facebook mining everyone’s social graph and so on.  Non obvious examples include artists mining the space of perceptively interesting things, doctors mining the space of symptoms, and businesses mining the space of sellable products and so on.

 

Let me consider in a little more detail that last one.  Every business is a program.  It’s a pattern (a pattern of patterns) interpreting the patterns closest to it (competition and the industry) and finding patterns for its customers (persons or government or companies or other patterns) to buy (currency is just patterns interpreted).   Perhaps before computers and the explosion of “digital information” it wasn’t so obvious this is what it is.  But now that so much of the world is now digital and electronic how many businesses actually deal with physical goods and paper money?  How many businesses have ever seen all their employees or customers?  How many businesses exist really only has brief “ideas”?   What are all these businesses if not simply patterns of information interpreted as “valuable?”.  And isn’t every business at this point basically coming down to how much data it can amass and interpret better/more efficiently than the competition? How are businesses funded other than algorithmic trading algorithms trading the stock market in high frequency making banks and VCs wealthy so their analysts can train their models to identify the next program, er, business to invest in…..

 

When you get down to it, everything is programming.  Everything we do in life, every experience is programming.  Patterns interpreting patterns.  

 

The implications of this are quite broad.   This is why I claim the next major “innovation” we all will really notice is an incredible leap in the capability of “programming languages”.   I don’t know exactly what they will look or feel like but as the general population desires to have more programmability of the world in a “digital” or what I call “abstract” way the programming languages will have to become patterns themselves that are generally more easily interpreted (written by anyone!).   The more the stuff we buy and sell is pure information (think of a future in which we’re all just trading software and 3d printer object designs (which is what industrial manufacturers basically do)) the more we all will not want to wait for someone else to reprogram the world around us, we all will want to do it.   Education, health care, transportation, living, etc. is all becoming more and more modular and interchangeable, like little chunks of programs (usually called libraries or plugins).   So all these things we traditionally think as “the real world” are actually becoming little patterns we swap in and out of.  Consider how many of you have taken an uber from your phone, stayed at an airbnb, order an eBook from amazon, sent digital happy birthday and so on…. Everything is becoming a symbolic representation more and more easily programmed to be just how we want.

 

And so this is why big data is all the rage.  Not because it’s a cool fad or some new tech thing… it’s because it’s the ONLY THING.   All of these “patterns” and “programs” I’m talking about taken on the whole are just the SPACE OF DATA for us to mine all the patterns.   The biggest program of all is EVERYTHING in EXISTENCE.  On a smaller scale the more complicated and complex a program is the more it looks indistinguishable from a huge pile of data.   The more clever we find our devices the more it turns out that it’s an inseparable entanglement of data and programs (think of the autospell on your phone… when it messes up your spelling it’s just the data it’s gleaned from you….).  Data = programs.  Programs = data.   Patterns = patterns.   Our world is becoming a giant abstract ball of data, sometimes we’re symbolizing but more and more often we’re able to directly compute (interpret natively/without translation) with objects as they exist (genetic modification, quantum computing, wetware, etc.).   In either case it’s all equivalent… only now we’re becoming aware of this equivalence if not in “mind” then in behavior or what we expect to be able to do.

 

Face it. You are a programmer.   and you are big data.

Read Full Post »

Anyone that has worked with me is tired of me suggested that everyone in business should know how to program.   This thought is met with a variety of rebuttals that have only a slight shred of a validity.

Everyone programs.  If you get out of bed in the morning and go through any sort of routine (everything is pretty much a routine) you are programming.   This is not semantics. Programming is nothing more than organizing things in such a way that they transform into other things.   Everyday life is programming, it’s just not the uber-formal (re: very restrictive) programming  we think computer programmers do.

When people reject my statement about everyone programs and should get better at what they are actually rejecting is the specific implementations of computer programming – the syntax, the formalities, the tools, the long hours in front of a headache inducing screen.

If you speak, write, draw or communicate at all you have learned a set of rules that you apply to various inputs and produce various outputs.   If you work in spreadsheets, at a cash register, with a paint brush, in a lecture haul, in a lab, on a stage, you are programming.   If you make yourself a sandwich, eat it and go for a jog, you are programming.  Everything you do is taking inputs and transforming it into outputs using various rules of a system.   The system is more or less formal, more or less open.

I don’t see there being any room for dispute on this observation or rather this definition or axiom.

With that basic assumption as a starting point let me make the case that honing your more formal, strict and, yes, traditional “computer” programming skill is a must do for anyone participating in modern society.  (yes, if you do not participate in modern society and do not wish to do so, you don’t need formal programming skill, but you will always be programming within the universe…)

Without getting too out there – our lives will never have fewer computers, fewer programs, fewer gadgets, fewer controllers monitoring, regulating, data exposing, recommending, and behaving on our behalf.   Cell phone penetration is near ubiquitous, every car has computers, trains run on computerized schedules, more than 50% of stocks are algorithmically trade, your money is banked electronically, the government spends your taxes electronically and so on.   So in some sense, to not be able to program formally leaves you without any knowledge of how these systems work or miswork.  Some will have the argument that “I don’t need to know how my car works to use it/benefit from it.”   This is true.  But computers and programming are so much more fundamental than your car.   To not be able to program is akin, at this point, to not being able to read or write.   You are 100% dependent on others in the world.  You can function without a working car.

Before you reject my claim outright consider the idea that learning to program is quite natural and dare I say, easy.   It requires no special knowledge or skill.  It requires only language acquisition skills and concentration which every human i’ve read about or know has these two basic capabilities (before we go on destroying them in college.)

Why do I make this claim of ease?

Programming languages and making programs that work rely on a very small language.  Very simple rules.   Very simple syntax.   Frustratingly simple!   The english language (or any spoken language) is so much more ridiculously complicated.

It does not surprise me that people think it’s hard.  It’s frustrating.  It’s the practice and the simplification of your thoughts into more simple languages and syntax that’s hard.   And so is writing a speech others will understand, or painting a masterpiece, or correctly building a financial accounting book, or pretty doing anything you do for a living that requires someone else to understand and use your output.

I firmly believe each persons ability to translate their lives into useful programs is a differentiator of those that have freedom and identity and those that do not.  Either you are programming and able to keep watch over the programs you use or you are programmed.

Sure, companies and people are busy at work making easier and easier tools to “program” but that doesn’t change the fundamental problem.   The programs you layer on top of other programs (web page builder guis to HTML to browser parsers to web servers…) the more chance of transcription problems (miscommunication), unnoticed malicious use and so forth.

Beyond the issue of freedom it is fun and invigorating to create, to mold your world.  This is the part that’s hard for adults.  Having spent probably from age 10 to whatever age we all are following rules (others programs) and being rewarded (program feedback loops) we all don’t really do a great job molding our world.  Kids are so good at experimenting (playing).   And playing is essential to really great programming.   Programming that will fill you up and make your life better is the kind that generates wonderfully unexpected but useful results.   It’s not always about getting it right or spitting out the answer (though for simple programs that might be the point).  It’s about creating, exploring, and finding connections in this world.

I can replace the word programmer (and programming) in this post with Artist, Mathematician, Reader, Writer, Actor, etc and it will be essentially the same piece with the same reasoning.   All of these “occupations” and their activities are programming – the only thing that differs are the implementations of language (syntax, medium, tools).

When people are rejecting my argument that everyone should learn to program, they are rejecting the notion of sitting down in front of a blinking cursor on a screen and having a piece of software say “error”.   Reject that!  I hate that too!  For me, correcting grammar in my posts or emails or journals is as painful! (but it doesn’t prevent me from wanting to write better or write at all, i *need* to to survive and be free!)

Don’t reject the notion that you shouldn’t be always trying to communicate or understand better – taking inputs from the world and transforming them into useful outputs.  To reject that is essentially rejecting everything.  (and that is now the annoying over-reaching philosophical close!)

Read Full Post »

Here’s a nice intro to functional programming on Dr. Dobbs.

In any functional programming language, you are likely to encounter these features:

* First-class functions, or higher-order functions: Functions can serve as arguments and results of functions.
* Recursion as the primary tool for iteration.
* Heavy use of pattern matching, although technically it is not a defining feature of FP.
* Lazy evaluation, which makes possible the creation of infinite sequences and other data structures.

At the end of the article on page 2 you get a nice discussion of Mathematica as a functional programming language.

It’s been noted a lot lately in technical publications that functional programming will continue to grow in importance due in large part to the need for parallel computing.  Functional programming is well suited for massive parrallel computation for a variety of reasons and the article does a good job highlighting some of those.  Of course, it comes at a price to developers – the learning curve.

It’s not a trivial exercise to switch from OO and imperative styles into functional.  And the lack of huge repositories of free code makes doing a quick commercial application fairly challenging.

Read Full Post »