Wednesday, December 31, 2008

Term Extraction Tools

Revisiting my long dormant toy that uses analysis of song lyrics and Flickr or Google Images to do contextual visualizations. I have implemented a simple ti.idf frequency counter and am getting decent results but wanting for more. In this web 2.0 world maybe I should be looking for services to do this?

Consider:

Yahoo Term Extractor API
Topicalizer
OpenCalais
Alchemy

and mabye Wordnet for synonyms.

Also, possibly relevant is Echonest's colossal db of music metadata intended to spur research into music recommendation / discovery tools.

Wednesday, December 17, 2008

Why we Iterate

The struggle continues. Faced with being increasingly outnumbered by PMI habituated and prediction minded project managers we've hired lately I'm returning to this question of why we iterate.

People are using the word. But too often not groking the motives.

We have things like 9 month long
efforts to build an internally deployed version of a currently external system being called an iteration. To me that's a whole other project or a "phase" but certainly not an iteration.

We have PM candidates that my peers are gushing about. I get asked to interview. I ask about things like how they think requirements work should progress in an iterative methodology. I get back ideas about getting tomes of requirements first and then planning iterations.

We have months long breadth first swipes through the requirements for a whole system being called an iteration along side plans for a second iteration to do "design".

We have business users insisting that it's pointless to test an incomplete system. "I dont want to see it until it's done".

We have directors asking for MS project plans in the first weeks of a project detailing how many iterations it will take and what will be done in each.

Seems like we have a future destined for many more failed projects and missed "estimates" before we start to figure this out.

I have dreams about some speech, presentation or as of yet un-conceived prop that will suddenly bring the idea home to folks. But in waking life am running thin on optimism that we can get this concept across and make it work.

So, maybe one more time I'll try to collect some nuggets of persuasion and contemporary thought about why emergent software planning is more healthy than predictive planning.

Maybe something will come of it.

Coding Horror: Boyd's Law of Iteration is a good summation of the detail in Roger Sessions' A better path to EA

The neglected practice of iteration Jeff Patton sends a reminder that software developers who neglect the practice of iteration will get caught either delivering poor quality software or delaying schedules in order to make time to iterate. We kick ourselves, or others, for not "getting [software] right up front" when we all know that the hardest part of software development is figuring out what to build. But there's hope, and it comes in the form of prototypes and frequent iterations

Good quip about devs not being the constraint From David Anderson's blog article Why We Lost Focus on Development Practices
In my Zen of Agile Management class, I teach participants about constraints. I then ask them, in a collaborative exercise, to speculate about the bottleneck in their organization and discuss how they would prove it and what they would do about it. In almost 3 years of teaching this class, over 4 continents, and around 12 occassions with groups ranging from 16 people to 250 people (at Javapolis in 2007), I have concluded that developers are the constraining factor on project and team performance less than 10% of the time. In some groups it is as low as 3%.

My belief around this is quite simple. Basic agile practices focusing on quality including collaborative working such as pair programming, and a strong focus on unit testing and early defect detection with continuous integration and test automation, have greatly improved software development to the point where initial software quality (i.e. bug insertion rates) is not the constraining factor on team performance. With immature teams, with sloppy development practices and poor initial quality (i.e. high defect insertion rates) development is the constraint. Agile has successfully fixed this!

So we can declare victory!

In this sense, the crowd who argue for a continued focus on better development practices are fighting the last war.

The rest of the community moved to other areas - most notably project management and business analysis / value-based software engineering. The community tends to get sucked to where the constraining problems are occurring. This is the natural and correct behavior. And it shows that many in the community interpret better ways of developing software in the broad sense rather than the narrow programming-centric sense.

Metaphors & Quips
OOPA - Observe, Orient, Plan, Act (or maybe P really should be Decide)
Shoot, Move, Communicate
Fail early and often.