Feeds:
Posts
Comments

Archive for December 24th, 2007

Summary (PDF of Draft Analysis)

What started for me as a typical “read Slashdot” for a minute has turned into a full blown research project into collaboration. The participation in solving the N-BRAIN Master Software Developer challenge delivered huge amounts of experiential as well as quantitative information regarding social collaboration on software projects.

This is a particularly good research situation because the stakes were reasonably high (potential job interview, Slashdot ego boost, public display of skill), the timeframe condensed, and the entire thing is trackable/auditable.

This blog post is the results of my findings so far (less than 12 hours after the solution to the challenge).

The Set Up

  • Unknown company posts a want ad on craigslist that includes an invitation to solve the challenge for a chance at an interview. Read here for launching point.
  • Full job posting here
  • Slashdot.org community picks it up quick and several developers/technical people set to work. Initially using Slashdot comments to post back and forth
  • The easy clues lead first to a Google Group, bringing together the challengers
  • The community forms of its own accord with no prodding or seeding (that we are aware of)
  • Google groups becomes repository of thoughts, questions, ideas, code samples, files, conversation, drawing board (please see for the final code samples and all that. Very impressive stuff)
  • Google groups tracks all contributions by login (handle), topic (community assigned), and datetime stamp
  • The Challenge urls
  • Background info for the layperson
  • Tools Used in Challenge
    • Programming Languages
      • Perl – character counts/frequencies encoder/decoder
      • Python – character counts/frequencies
      • Java – for encoder/decoder
      • Piet (npiet)
    • Software
      • Photoshop (to count pixels)
      • Npiet (for test analysis)
    • Sites
      • WhoIs.net
      • NetworkSolutions
      • Craigslist
      • SlashDot
      • Google Groups
      • Wikipedia
      • TinyURL
    • Historical Figures and Places and Times
      • Henry Ford
      • Samuel Smiles
      • Charles Buxton Going
      • Boulder
      • Servus
      • Flavian II
      • Turing
      • Van Gogh
    • Processes/Techniques (list from PeterOfOz, contributor)
      • Game playing (recognizing a Tetris like pattern)
      • Javascript, Perl, Python, and Java programming (probably others as well)
      • Knowing how to inspect HTML pages, and includes for javascript and
      • CSS
      • Web research (finding the original Ford passage, Pi lookups, Latintranslations, etc)
      • Lateral thinking and pattern analysis/recognition
      • Cryptographic analysis
      • Graphic formats
      • Numerical sequences (pi)
      • Byte code engines
      • Encoding/decoding engines

Questions

This analysis focuses on several questions:

  • Quantitative
    • How quickly was the problem solved
    • Relative percentages of general contributions to key contributions
    • Distribution of contributions over time and by person
    • Classification of contributions
  • Qualitative
    • Can a group solve things faster than a really talented individual (yes! We squeezed in 400 manhours in 18 real hours)
    • Is there any correlation between quantity and quality (hard to tell. This was a complicated challenge and the solution didn’t need to be anything more than a one off solution.)
    • Are there biases by contributor (80/20 rule, is 80% of the work done by 20% of the people) (yes! But different levels. Breakthroughs supplied by handful of people, grunt research supplied by another group.)
    • What makes a successful collaboration (solving the problem, of course! but doing it with fewer errors, better documentation, on time, on budget.)
    • What didn’t work (redundant work on encoder/decoder, multiple threads going at once, timezone differences without known “schedules” kept folks out of sync at the end… would improving these speed up this solution? improve its quality?)
    • What were some of the group dynamics (more to come on this in later posts… roles people filled…)
    • What schedules of reinforcement were at play (more to come on this… the feedback loop of the group and how code/solutions become reinforcers)
  • What I wish I had access to (Companies if you are reading this, please provide it will be WORTH IT FOR ME TO ANALYZE IN TERMS OF GOODWILL AND PUBLICITY. UPDATE 12/24 morning: N-Brain reached out to collaborate!)
    • Traffic Logs from Google
    • N-Brain (company behind it) assumptions
    • Traffic logs on N-Brain
    • Interviewees Invited
  • Follow Up Analysis (will follow up in January or sooner)
    • Traffic generated to the end site, n-brain.net (can tell in quantcast.com, compete.com, and alexa)
    • Traffic generated to http://wanted-master-software-developers.com/
    • Profiles of the contributors (get resumes/cvs/bios and/or some basic demographics)
    • Success of N Brain Product Release

The Analysis

Key observations

There was almost NO FLAME WARS/NEGATIVE COMMENTS AT ALL

Very little correlation to posting frequency/amount and breakthrough chance (biggest breakthroughs produced/cited by some of the least frequent posters)

Key Facts

Dataset

Over 600+ postings, 300+ real contributions, 25 breakthrus (less than 10% of contributions were breakthrus)

Took 18 hours and 132 people (73 contributors, 59 observers) to solve challenge.

No Slashdot comments were included in this analysis. It should be noted that several key findings appeared there first. The main finding being the google group to launch the real challenge. Many of the key postings on Slashdot were made by persons who migrated to google group, so it should not affect analysis too much.

Workload

Estimation that approximately 19 people put in 10+ hours. Approximately 400 man hours put in, with more than half by 19 people (analysis adjusted for sleeping time and by timing of contribution. E.g. if contributor had to sleep, discount 7 hours)

 

5.65 contributions per person. Max contribution count was 25. Minimum was 1. Most people contributed less than 5 times. It should be noted 3 of the key breakthroughs came from contributors with fewer than 5 contributions.

Peak activity and Peak breakthroughs not correlated

Classification of Workload

(classifications subjective to analyst. Probably could use a second eye)

Most of the contributions were research or clues. A lot of research chased down dead ends or irrelevant facts. Very little banter or small talk. No flames on Group. A few on Slashdot.

Breakdown of contribution classifications by Contributor.

Note: the data has been scrubbed for contributions/postings that weren’t mere banter or blank. (I full admit to likely misclassifying and even misassigning breakthrus and solutions to contributors. Please correct me if I did.)

Note: I considered breakthroughs as contributions that were sub solutions, code implementations that lead somewhere or key insights into clues.

Please SEE PDF FOR TABLE ON CONTRIBUTOR BREAKDOWN
(PDF of Draft Analysis)

Conclusion

N-brain got more than their money’s worth for creating this test. Beyond uncovering great talent, they learned a lot about collaborative development, especially in a wide open problem set.

Open style collaboration is incredibly efficient. We squeezed in 400 manhours into an 18 hour period on a holiday weekend.

There’s room for all types. Almost all contribution behavior that HELPED was quickly reinforced (follow up analysis of feedback loop to follow). Anything that was redherring or slightly counter productive was extinguished almost immediately. We had one instance of information withholding early on that was quickly eliminated and never resurfaced.

Tracking of projects happens quite naturally now with all our web based toolsets. No disruption of creativity or coding occurred and we have a fully analyzable project.

We need to analyze more of these situations to give businesses, organizations and individuals a strategy for existing in this flat global world. More on this later…

What do you conclude?

~Russ

Read Full Post »