NDP Software

Getting paid by LOCs?

2012-01-16T21:36:00.000-08:00

I don't get paid by the lines of code I write. But I've certainly joked about it: there's a rumor that Trevor, who writes way too many comments, gets paid by the line (including comments). Über-programmer Dave things his manhood is measured by the size of his commits. On a recent project the codebase was so large that I joked that every commit should be rejected if it didn't reduce the line count of the project. Haha.

This brings me to the boy scout rule: always leave the code nicer than when you started. This can be a little hard to quantify with a number: test coverage, cyclomatic complexity and coupling metrics are all often used. But with a bloated codebase, a simple line count reduction may suffice to improve the codebase, although is not the main goal. In my casual joking about this with my teammates, they all liked it. But how?

How to Count

If you use git log --stat, you can see the number of lines associated with each commit (after all the details), such as:

331 files changed, 0 insertions(+), 25866 deletions(-)

It's pretty easy to write a script that parses this information out, and that's what I've done in gitlc tool. Beyond the obvious grep, I added support for some necessities:

"pairs" set up using the git-pair tool. Commits checked in by multiple people will be credited to all committers. This is necessary with any project where pairing is used.
"aliases", so that if the same person commits with different email addresses (or it changes throughout the lifecycle of the project), the commit counts can be aggregated. This optional feature can be used by providing an optional yaml file.
commits can be aggregated by person, month, or simply by commit

Usage is pretty simple. Just download it and point it at a local directory with a .git folder. Here's gitlc source itself and ask for a summary by person:

$ ./gitlc.rb -r . -p

[["ndp", {:net=>657, :adds=>26579, :deletes=>25922}]]

As you can see, I had a 25K of code, which I abandoned quickly (thanks node). For projects with more people, it's more interesting:

./gitlc.rb -p -r ../ruby-build/

[["sam", {:net=>499, :adds=>639, :deletes=>140}],

["jeremy", {:net=>33, :adds=>50, :deletes=>17}],

["josh", {:net=>21, :adds=>41, :deletes=>20}],

["jesse", {:net=>16, :adds=>21, :deletes=>5}],

["guilleig", {:net=>11, :adds=>53, :deletes=>42}],

["bensie", {:net=>6, :adds=>6, :deletes=>0}],

["chris", {:net=>6, :adds=>29, :deletes=>23}],

["sstephenson", {:net=>5, :adds=>5, :deletes=>0}],

...

The tool sorts people by their "net" contribution to the project. For ruby-build, there's a long-tail of people with 1 line changes omitted. Please check out gitlc and see what you learn about your project.

I'd love to have contributes to this tool. Feel free to fork and contribute back. It's easy to imagine better visualizations and a tool that could be incorporated in a workflow.

Practicalities

When you've got these statistics, and people start to care about their line count number. People ask me what to do if you need to write new functionality?

I recommend:

Look first to share code. In a relatively large codebase, most features should already be there, but perhaps in a slightly different form. Most codebases bloat because we aren't able to figure out the common patterns. We can blame this on a hazy product definition, but at some point it's our responsibility to organize and structure the code ourselves.
Look for dead code. What features and code are no longer used? Get code coverage tools running, which help you identify branches of code that you can remove. Use web analytics to discover pages that are seldom visited and advocate to cut them out.

Riffing on the last point, a corollary to this blog post is the one that says each story needs to be accompanied by an "unstory". If we add a link to this page, what link do you want to take away? If the user can now spend 20 minutes cropping their profile picture, what aren't they going to be doing? But that's left for another day...

Do I really have to write tests?

2011-05-05T21:54:00.000-07:00

Over at the Lean Startup Circle mailing list, they're discussing what kind of tests to expect from developers. I enjoy this conversation. People are often looking for a clear guideline ("startups don't need tests") or "code coverage" figure-- or have one in mind. Idealistic agilists insist you always write tests. They equate not writing tests with abandoning quality-- the beginning of the end. "Such carelessness will lead to bugs upon bugs and eventually, unmaintainable code." Reality requires a more nuanced and pragmatic approach suggested below.

Getting Real

I learned in Q.A. school (or was it the Q.A. streets?), you can't test everything. Be pragmatic because software is just too complex to allow coverage of all cases. Good test design requires thoughtful prioritization-- minimal money yielding maximum test coverage. Q.A. engineers learn the requirements and the relative importance of each. They let this understanding guide how you design a test suite. They learn to carefully consider the quality requirements, where the critical parts of the software is, and where potential problems might appear, and what the costs of failure are. In this cost-benefit and risk analysis, there isn't just one, simple answer.

The main factors to consider to judge how much testing is needed, keeping an eye towards developer-written tests:

Cost of bugs. Different software has different quality requirements. One of my projects would cost $1000s of dollars per hour if it crashed-- and cause freeway backups, and bring the L.A. news helicopters, etc. Another one let you comment on your latest book purchase. Some applications really are heart-lung machines.

Anticipated life of the code. Some code is for prototypes or other "throwaway" purpose. But veterans bristle at the thought, and will recount how quick hacks became the core of mission critical systems, and that code just doesn't go away. Seconding their motion, TDD disciples insist on writing tests for everything. There certainly is risk in creating code too casually, but prototyping is a powerful tool. Prototyping is powerful. If you can figure out how to build out your ideas quickly and explore them with your users, you'll be able to able to out-pace competitors.

Area of the application. Not all code of an app is the same. At a concert, there are soloists and choir members. The audience will be much more critical of the soloists. Singing coaches don't spend as much time with each choir member as they do with soloists. Therefore, identify the most important part of your app.

I'm a big advocate of building from the inside out: identify these critical parts of the system. Validate the model with everyone, and test the heck out of them. Make it unbreakable. Then, using these as building blocks, put together the pieces in various ways. Allow yourself to test different combinations with the knowledge that the blocks won't break from underneath you. Many a team has treated speculative admin interfaces with the same vigilance as the core data model. It's quick to build prototypes with solid building blocks.

Likelihood of bugs. Some areas of an app are like compost piles, and attract bugs. Good developers should be able to give you an honest appraisal. They should be able to identify difficult problems, as well as error-prone approaches to problems. Some code just evolves to become gnarly to work with. Adjust testing needs appropriately.

Test-driven design requirements. TDD is an excellent tool for figuring how the code should be written and structured. I find that using TDD, when informing the design, save significant development effort. Don't sacrifice testing when it will benefit the code. If it's faster to write a prototype with TDD, I'll use it.

Team size. A lone developer focusing on one project will have less of a need for tests than a larger team all sharing a codebase. Yes, with luck, the project will grow into something larger, but there's also an important "just-in-time" theme to agile. This is not an excuse for skipping testing, but is a factor to consider.

Other quality checks. A team auto-deploying to the cloud needs more checks-- and tests-- than a team with a dedicated Q.A. team watching out for them.

There are other factors, but those are the important ones. Let's resist talking in absolutes, and do what is best for our project, customers and users.

Interesting Character Entities

2011-05-03T20:09:00.000-07:00

I spent another couple hours on my character entity finder, and wanted to share some of the interesting things I've discovered. I made two main improvements:
(1) lookup happens asynchronously, in small batches, so that it suffers less from "locking up". This allows much more flexible exploration.
(2) allow you to bookmark queries, so you can share interesting discoveries.

To recap, I built this tool to find all the weird quotes:

http://ndpsoftware.com/&what/#q='
http://ndpsoftware.com/&what/#q=quot

Wow, I discovered there's a nice set of chess pieces:

http://ndpsoftware.com/&what/#q=chess

Just a minute ago, I added the ability to bypass the query builder-- if you enter a full regular expression. The regular expression is matched against entity names (and numbers, and nicknames). For example, there are all sorts of icons available:

http://ndpsoftware.com/&what/#q=/97[2-6]\d/

The currency symbols aren't easily found with a single query, but you can build a page with a selection of them:

http://ndpsoftware.com/&what/#q=/currency|euro|dollar|pound/

It's surprisingly fun to play around with. Give it a spin.

Finding entities, characters, glyphs

2011-04-04T23:43:00.000-07:00

My latest "mini-project", a few hours in, solves the annoying problem of trying to remember entity numbers or names. For example, our last project used the entity » (»), and it seemed like it took me two years to remember it. Now, with this tool, I can just type ">>" and the character, symbol and number appear. I tried to make it "mobile friendly", and may experiment with packaging it as an "app".

I'm intrigued by "mini-projects"... something I can put together in a few minutes or hours and provide value to someone. It's a long-time obsession: in the 90s when I got my first laptop, I tested myself to see what I put together on my 22 minute BART ride from Oakland to 24th Street. It was fun, but I never creating anything of general interest.

Now, these little tools (and hacks) can be quicker to build out and easier to share. In fact, there are 1000s of "apps" out there, many of which look like they can be done in hours. Anyway, I have a few on my web site.

What makes things easier is there is so much data out there to build upon. As you'd expect, it takes the entity list of W3C. But I supplemented this with interesting Unicode characters, and a nice set from Remy Sharp.

Check it out, bookmark it, and please send me feedback.

Friendlier Session Timeouts 2.0

2011-03-28T21:44:00.000-07:00

As we discovered today, coding up friendly session time-outs involves more than meets the eye. As you know, a session time-out logs the user out after a period of inactivity. But interactions with web sites, and "inactivity," have changed over the last 10-15 years.

We have a fairly plain Rails app, and we're implementing a series of security fixes in anticipation of an audit. In what seemed like a simple "1-pointer", we were asked to remove the "Remember Me" checkbox from our application, and force the user to be logged out after 20 minutes.

For the first pass, we simply set the session timeout to 20 minutes and removed the checkbox (and the underlying implementation). Easy eh?

Sadly, this solution leaves much to be desired. When the user (or Q.A. engineer) returns after 1/2 hour, the page is unchanged and ready for action. But any click redirects, confuses, and potentially loses in-progress work.

Although this might have be acceptable in 1998, it's 2011 and we got Ajax. Clicking a link on a Web 1.0 site takes you to another page; if you happen to be logged out, oh well, sign back in and continue-- you probably weren't doing something that important anyhow.

But these days, on an Ajax page, if your session expires while you're viewing the page, clicking on any element of the page can reveal a session timeout. This may appear to the user as a server error, or if it's handled correctly (like we did), a page redirect. Clicking on a disclosure triangle redirects to a new page? Now we have a surprised user.

I Googled a bit and found lots of bug reports around this behavior. As this Jira snapshot shows, resolving this might be trickier than anticipated.

I checked out my bank's solution. It looked like they implemented a whole timeout scheme in Javascript. Were they wacko? (No... but we'll get there.)

Of the hand full of ideas out there on the web, here's one solution (of a hand full) with the idea of timing the session in Javascript:


<script language="javascript">

function confirmLogoff() {
  if (confirm("Your session will end in one minute.\n\nPress OK to continue for another ten minutes.")){
  location.reload();
  }
}
setTimeout("confirmLogoff()", <%= sessionTimeout - 1000*60 %>);
</script>

At first blush this looks like they might be on to something. Alas no. The confirmation will only work the brief period between the client and the server timeout. I'd argue it's even worse than no solution, since the messages promises something it can't deliver on, and loses the user's work-in-progress.

Did I mention we have quite a few pages with AJAX requests on them? These conveniently extend the session timeout on the server when the user interacts with them. But they break any sort of client-side timer that is set up at page load. Any sort of "meta refresh" schemes doing something similar to the above were quickly dismissed.

And why should just AJAX backed behaviors restart this timer? Opening a hidden panel may or may not go to the server (depending on an developer's whim), but should this whim this really affect the user's timeout? So we started considering implementations that ping the server as the user interacts. This was also dismissed as being complicated to implement efficiently (and potentially introducing some sort of security issue).

Finally where we "settled" (as in prom date), is implementing a timeout within Javascript, like my bank. It's a little more sophisticated: it's reset not only by the initial page load, but all sorts of user interactions. The code finally reduced down to:


var clientSessionTimeout = function(timeoutMS, logoutFn) {
  var lastTimeout;

  var startSessionTimeout = function() {
    if (lastTimeout) clearTimeout(lastTimeout);
    lastTimeout = setTimeout(function() {
          logoutFn();
      }, timeoutMS);
  };

  // Watch for activity
  $('body').click(startSessionTimeout).keydown(startSessionTimeout);
  startSessionTimeout();
};

This is called with the 20-minute timeout, and implements it beautifully:


    clientSessionTimeout(20 * 60 * 1000, function() {
      document.location = '/timeout?return_to=' + document.location.href;
    });

The server side timeout is for redundancy. Our pages are all pretty focused, so I doubt any user will spend more than a couple minutes on any one of them. We picked a server-side session timeout of 40 minutes. The only way the Javascript timeout won't kick in first is if the user interacts with the page for more than this time with no server side interaction... possible, but not likely.

After completing this compromise solution, I'm ready to spell out some ideal requirements:
* session timeout with no interaction after 20 minutes
* any interaction on the page should reset the timeout
* warn the user (if possible) when the deadline approaches
* this shouldn't open additional security vulnerabilities or server traffic

With some additional work, I'm sure an "ideal" solution can be developed. This compromise should get us most of the way there. Thanks to the rest of my team, and an interview candidate who provided some clear thinking on the matter.

Curing Frequent Selenium File Upload Failures

2011-03-18T19:14:00.000-07:00

The symptom was quite simple: do an upload, and on the next request the server reports an "IOError". As our Ruby on Rails app is pretty much thin workflow around lots of file uploads, this was a problem. We tended not to see in on production, but us frequent users were seeing it enough to know we had to do something about it.

But the real complainer was Selenium. About half the time the tests failed and needed to be coaxed into running again.

JWinky traced the root cause down to a known bug in the temp file class. With a little work (and encouragement by yours truly), he put together a patch that has eliminated the problem. We've been running with it for a couple months and haven't seen the bug once-- or heard a peep from Selenium.

It's found here: http://github.com/jwinky/ruby_tempfile_ioerror

Updated Cheat Sheets

2011-03-01T21:28:00.000-08:00

Five years ago, I was working at Great Schools, and got interested in SEO. I started running all sorts of experiments on my own site, to understand how I could affect things. I reorganized the URLs, added keywords, and followed all the standard recommendations. I quickly realized little tweaks to URLs, meta tags, and optimizing keyword density wasn't going to help much. These types of changes really are "optimizations"-- they'll give you a small percentage increase, but they are not game changers. If you've got millions of visitors, a 1% may mean real money, but if you're me, it doesn't matter.

So, after working on SEO, I pursued another idea. Why not create something of real value to drive people to my site? I had an idea and created some "cheat sheets" to help me with my own development. I created them, and then posted them where I could to get some inbound links. Shortly thereafter, someone at O'Reilly found my page and linked to it, and all of a sudden I was getting hundreds of page views per day. So that was my lesson: If I provide something of value, people will come. That was five years ago. Even though technology changes fast, I still have a bit of tail from those original cheat sheets.

Last night, I decided that since of 80% of the people hitting my site are seeing those pages, I should take a look at them and see what impression they might be making. I really don't have a "goal" of driving traffic anywhere else, but I might as well make them look as good as I can. So I cleaned up the visual design and fixed some of the editing.

Check out the spruced up pages here: Hibernate Mapping and JSPx Cheatsheet.

Color Scheming with Javascript

2011-01-21T08:09:00.000-08:00

In this post, I'll share some of my Javascript code for manipuating colors.

I started a few months back with basic color manipulation routes. Other libraries take a strictly object-oriented approach. This can be a little heavyweight, as it requires explicit conversions throughout the calling code. But in the HTML DOM, colors are generally expressed as hex strings, and if we have routines were built around these, they would be simpler to use. Plus, Javascript is dynamic language, so a String could have color manipulation methods. That's exactly what I did:

'#fff'.toHexColor() => '#ffffff'
'black'.toHexColor() => '#000000'
'#123456'.toHexColor() => '#123456' (no op)
colorString.toRGB() => array of numbers [0..255]
colorString.toHSL() => array of numbers [[0..360],[0-100],[0-100]]

These building blocks aren't that exciting, but are very helpful to build color manipulation functions on:

"#ab342c".darken(%) -- make color darker by given percent
"#ab342c".lighten(%) -- make color lighter by given percent
"#ab342c".saturate(%) -- make color more saturated by given percent. To desaturate, use negative values for the percent. Note that"#ab342c".saturate(-100) renders in grayscale.

Generating new colors
Sometimes, you just need a color to get you started:

ColorFactory.random() // a random color, somewhat evenly distributed.
ColorFactory.randomGray() // a random gray scale value.
ColorFactory.randomHue() // given a saturation and lightness returns a random color.

or primary and secondary color schemes:

Basic color theory includes the concepts of complementary and analogous colors, so they are provided:

And there is a generic "interpolation" beween two colors. It works interpolating hues, saturation or lightness:

Generating Schemes for Visualizations
Now that we have all the building blocks, we can start building schemes for specific purposes. (Schemes get applied in specific ways). For example, if colors are used to represent a quantitative range of values, the colors must visually read as such. The most straightforward way to do this is to linearly lay them out in a monochromatic scheme. Your eye can read "that value is more than that value" because of the visual relationship, in either saturation or lightness.

This differs from qualitatively distinct values (such as states on a modern election map), which must be read as distinct, but not qualitatively related. A triadic approach is more appropriate in this case. A viewer should at no point be enticed into imaging a "red" state is more or less of something than a "blue" state-- they are distinct categories.

Sometimes visualizations include binary, yes/no values. And some data visualizations are about how the values diverge, where values need to read from as quantitatively diverging from a central value. These are challenging to construct so that they read correctly, and cartographers are experts at this.

I've in the process of building a set up Javascript functions that support this type of color scheme generation. It's still a work in progress, but I have found it quite valuable. I'm curious if others find this useful.

Color Factory: Color scheme generators

2010-12-29T07:50:00.000-08:00

While working on a spin-off from bedsider, I created Csster. Alex @ C5 encouraged me (and coached me a bit) on getting through functions around color math, and as far as I know, the functions in Csster are some of the only Javascript implementations. I find the invaluable as I build out Javascript functionality, and I am working on separating them out from Csster itself.

Building my own visualizations, I've needed to bone-up on color theory. I thought I'd share some of what I've learned.

I've always been a interested in color-- mixing paints, collecting "color" items as a child, and design class in college. I've delved in deep in the past and have a good understanding from a theory of it. Most recently I caught up on the computational side of it, writing a Javascript library to help manage colors. To be honest, I still lack the practical experience necessary to have a great intuitive feel for colors, but I'm getting there.

Schemes & Tools

There are numerous software tools out on the web, and available for download, for creating colors schemes. Most are general free-for-alls-- paste in some hex codes please-- and offer no assistance, beyond live preview, tagging, named schemes or Google ads. The best of the bunch is probably the flashy Kuler.

The ones I like better (like Color Scheme Designer) bring to the forefront categories stemming from color theory and a color wheel, selecting a set of colors from a single tangent (monochrome), opposite sides (complementary), analogic (or analogous, for nearby). There are also triadic (3) and a host of variations and combinations of these. It's this type of color theory that designers study and provides the necessary grounding to understand a design and a "scheme" that supports it.

As I poke around building more visualizations, I quickly found that I needed "color schemes"-- not so much color manipulation as coherent sets of colors. I started building some of my own tools, and then stopped, and stepped back to understand some of the theory behind color schemes. One of the easiest from Cynthia Brewer here. Ms. Brewer explains a few different types of color schemes specific to data visualization: binary, qualitative, linear, and divergent.

Sequential schemes are suited to ordered data that progress from low to high. Lightness steps dominate the look of these schemes, with light colors for low data values to dark colors for high data values.
Diverging schemes put equal emphasis on mid-range critical values and extremes at both ends of the data range. The critical class or break in the middle of the legend is emphasized with light colors and low and high extremes are emphasized with dark colors that have contrasting hues.
Qualitative schemes do not imply magnitude differences between legend classes, and hues are used to create the primary visual differences between classes. Qualitative schemes are best suited to representing nominal or categorical data.

Using these schemes could bring sanity to lots of charting libraries out there-- and it's quite sad that they don't support such technology.

I'm building a "color factory" to provide functions for these. Provide the functions one (or two) colors from your palette and get a cohesive "color scheme" that should work with it-- and even better-- be appropriate to the data. These are all found in what I call the color factory. It's nascent technology, and I'm interested in feedback.

In the next post, I'll dig into the Javascript functions that facilitate this. (All this is part of a generic Javascript sketchbook.)

jQuery support in Csster

2010-12-23T21:57:00.000-08:00

I finally got around to adding a little jQuery plugin for my Csster tool, and released it as version 0.9.2.

$('.selector').csster({ width: 100 });

This looks a lot like the "css" method:

$('.selector').css({ width: 100 });

This difference is that Csster creates a "rule" and inserts it into the head, whereas jQuery will attach styles directly to the nodes. Sometimes you want one, and sometimes another.

It's convenient to use it in the midst of jQuery work, such as:

$('.sidebar').wrapAll('<div>').addClass('ready').csster({backgroundColor: '#ffeedd'});

It's also allows the "nesting" of regular Csster:

$('.hd').csster({ ul: { margin: 10, li: { margin: 0, padding: 5 }}});

Although it warrants its own post, I added a little note about how to implement clean browser-compatible patches. In the example, csster supports the "opacity" property name in IE by writing a simple Csster plugin that run only within the IE environment and applies the patch. Much nicer than subtler raw CSS solutions... more to come.

Check it out: http://github.com/ndp/csster

Learning Git?

2010-11-26T08:20:00.000-08:00

I just revised my previous visualization about git with an eye towards better visual design and usability.

Here's a little history: As I dove into learning git, I was initially confused about where my code was. I felt pretty confident that git hadn't lost anything, but less confident I could get it back readily. Sure, it's distributed, so I expect my code will be more places. But there was also this "index" and "stash"-- how do those relate? It's a little complex coming from Subversion or CVS.

Once I figured out the basic locations that things could be, understanding the commands is a second challenge. The commands tend to work on one or two targets, moving code from one to the other. But they aren't named in any obvious way, except for the "stash" commands. To make sense of these, I mapped them onto the locations. In the visualization, just click on "remote repository" to see all the commands that affect it.

Out of these two frustrations comes my visualization.

Intermittent Selenium Failures

2010-11-19T20:34:00.000-08:00

Selenium testing is always a little flakey, but I've* found a good treatment for this on my last two projects. It's pretty simple, really:

If you are using external Javascript services, turn them off.

This includes Google Analytics, Kiss Metrics, Share This, etc. The number of these services has exploded in the last couple years, and it's hard to build a site that doesn't use at least a couple. These tools do what they can to not interfere, but in the fast-paced world of Selenium, they don't always survive. Just remove them for these tests and you'll see marked improvement.

Actually, that reminds me of a good talk I heard the other night. It was by Marcus Westin and Martin Hunt of Meebo, and they talked about developing he "Meebo Bar". They figured out some really cool tricks to load asynchronously and not interfere with the host website-- but even better, supporting security contexts client side (which is pretty nifty if you think about it.) I actually think you could build a pretty clever SSO (single sign on) solution using these patterns, but I haven't tackled that one yet. Check the slides and presentation. A must read if you're developing your own widget.

* Actually, credit where credit's due: it was Justin and Jonah (different companies, different projects, not brothers) who identified this problem, not me.

Introducing Csster

2010-09-25T16:52:00.000-07:00

So I'm a bit of a CSS nerd. For years, I've been complaining that there's not enough "engineer" cycles given to CSS. I've written endless blog posts about how to organize your CSS. Blank stares when I ask interview candidates "how do you structure your CSS?" Well, now we can write CSS in Javascript with Csster, and maybe-- just maybe-- the world has been set right.

In 2000, I ventured into teaching at City College. To be honest, I'd missed the dot com boom and didn't know anything about the web. I was really nervous, so checked out every book the school library had about web development, and studied (and learned) thoroughly. Since then, I built quite a few web sites for people, non-profits and businesses, and developed a certain amount of dexterity with CSS. CSS has evolved a bit in the last ten years, but this firm foundation of understanding of the cascade and such has served me well.

Not to go off into a rant, but lots of CSS ends up a mess. I always felt like it, like C++, needed more restrictions and guidelines to turn out maintainable. I tried to enforce various guidelines with my teams.

In the last year, there's been a renewed interested in CSS and the tooling around it. With the advent of tools like SASS, it's starting to receive the engineering attention it's always deserved.

I was a little hesitant to endorse Sass (or even try it). CSS already has too many "degrees of freedom"-- more flexibility seemed like a bad idea. It would make the problem worse. I poo-pooed it.

Okay, stop laughing.

Finally a couple weeks ago with some changes to my team, I was convinced to give it another shot. At the same time I started a new small project, so I just tried it out. This project was design and Javascript heavy, but small enough to back out if it wasn't working. Sass was OK, I decided, but I was still maintaining too much junk, and now there was this additional build step.

On a Friday afternoon, I mentioned to my coworker Jeremy that what I'd really like is to just stay in Javascript. Javascript has this object literal format (JSON), and the whole hierarchy of CSS might just fit in.

It was a bit of an idea that I just throw out, but over the weekend I pursued a spike implementation of what it would look like. Using test driven design, it came together much faster than I had anticipated. It was a usable tool. Alex at C5 coached me through fixing some of the color functions, and as near as I can tell it equals or exceeds Sass's feature set.

In the last couple weeks I've used it extensively in my current project and it's great. Being in a real programming language really does make CSS nice.

Check out the main project page on github.

2010-09-19T10:15:00.000-07:00

Most resumes are pretty boring in their presentation. Textual descriptions of each job, reverse chronological. Every once in a while you find a programmer who "gets creative", rendering their whole resume in C++ syntax or a mindmap. We got one the other day that-- well, I'm not sure what it was, but it managed to communicate you'd never want that person working on a user interface.

I started to wonder what visualizations would be useful to a reader of a resume.
Although I think linkedin does a pretty fine job, here was one idea I sketched out with my new library Csster. I took some inspiration with recent work I've been doing with IDEO, as well as a summer of being surrounded by modern art in the Baltics.

Primary Programming Languages

What do you think? Is it interesting enough to explore the data?

If you had one visualization to complement your resume, what would you want to convey?

jQuery Conf SF

2010-04-19T21:50:00.000-07:00

After traveling around the Bay Area talking about Javascript Unit testing, I scored a shot at the San Francisco jQuery Conference. I'll be there Sunday afternoon, talking about "Organizing Your Code with Testable jQuery Plugins". Stop by and say "Hi"

Rails Fixtures with Integrity & Validity

2010-03-06T14:06:00.000-08:00

A new developer on the project changed the symbolic name of one fixture record and broke a whole bunch of tests in unexpected ways. Pairing, we discovered a some interesting stuff.

First, if you've never dug into them, it's critical to understand how symbollically named fixtures work. We rely on them heavily, but only yesterday read the code. If you have a fixture like:
bill:

full_name: ...

And another fixture:

socks:

owner: bill

Rails magically inserts bill's ID into socks' record. (Before this feature, developers had to manually manage their IDs and keeping fixtures working well was less fun.)

Nifty. I assumed (incorrectly), that there was some sort of lookup of records involved. So if I change the name of the "bill" fixture to something else-- let's say "william", I expect Rails to complain. It doesn't. There's no data integrity to the fixture system-- at all.

We traced though the code and now understand why: When Rails comes across the symbol "bill", it creates an integer hash of it and sticks it into the id column. It'll be a big number like 39384022 or something. Well, if you change the name to "william", it generates a different hash. That's it. At no time does it go back to verify that such a hash exists. It's really just a nice name for a number! Unless your database has constraints that enforce this (which will make loading fixtures more difficult and generally isn't done), you won't see a problem until a test fails.

Once we discovered that, we asked, "wouldn't be nice if there was a test that checked the integrity of the fixtures?" With a little work, we had just such a test, which relies on the validation system:
describe "fixture integrity" do

ActiveRecord::Base.send(:subclasses).each do |cls|

it "each fixture for class #{cls} should be valid" do

cls.find_each do |record|

record.valid?.should(equal(true),"Invalid fixture for #{cls}:\n #{record.errors.full_messages.join("\n")}\n#{record.inspect}")
end

end

Unfortunately that passed until we added a the appropriate validator
class Child < ActiveRecord::Base

validates_presence_of :parent

...

(Make sure you validate "parent", not "parent_id". "parent_id" will be set-- it just won't point to anything.)

That's it. A simple test and you won't risk your fixtures floating too far from the data structure you expect. Thanks to Lowell Kirsh and Jonah for pairing on this.

Implement Most Popular the Easy Way (hint: use Google Analytics, garb and Rails)

2010-02-06T21:22:00.000-08:00

Over the last few years I've implemented "most popular" posts, questions, lists, companies, users, pages, searches, cities, and who knows what else. It's not difficult. I've always implemented this myself-- using a few columns in an SQL database, but found something didn't smell right. We already have this free tool-- Google Analytics (GA)-- which is collecting usage data on my site. Why would I want to store this data redundantly?

In this post, I'll walk you through what we did on my most recent project for the National Campaign. There are three steps: collecting the raw data, processing it into statistics, and displaying it to the user.

Using GA you can completely outsource the data collection. For the statistical analysis, you gain flexibility-- more on that later. Finally, I won't talk at all about displaying the results to users-- that's up to you.

Step 1: Collecting the Data

If you are already using GA, you're done-- you're collecting data. If not, you simply need to create an account and start using it.

Fortunately, using RESTful conventions, most using actions end up being "page views" of some sort. But there might be other steps you want to take. For example, we had a page that served up content via Ajax, and I hadn't bothered to instrument them with GA yet. I added one line of Javascript to the Ajax callback: pageTracker._trackPageview(questionLink); And it can get more complicated: if your definition of popularity involves something beyond your pages, you'll have to dive into the event tracking or custom variables of GA (which I haven't done).

It's worth pointing out that if you don't use GA on a project, you need figure out what data to collect and how to store it. This involves the business owners expressing their requirements, and the developers debating which database table to use and how general a solution to build. You can imagine this can be a small sink-hole if you're not careful.

Step 2: Processing the Statistics

The hard problem to solve here is to create a function that calls out to GA, collects the data, and saves it into your database.

Understanding the API
Although you really don't need a deep understanding of the API, you should at least skim it. The salient points are to understand the differences between accounts, profiles and sites. Run off and read it now.

Install the Rails Gem
You'll need to get "garb"-- not as in trash, but as in Google Analytics for Ruby. Install it:

  gem install garb

  config.gem 'garb' # in environment.rb

rake gems:install

Profile Access

  require 'garb'

  def self.create_profile(acct, username, password)

    Garb::Session.login(username, password)

    Garb::Profile.first(acct)

end

Retrieving the Data

To retrieve the data, you generate a report. For this, you'll need:

dimensions: I just asked for page_path. If you ask for just one dimension, you can think of it as the row headers in the table you get back.
metrics: I just asked for the pageviews-- it's the values that go in the row cells returned. You can also ask for bounces, visits, entrances, exits, etc. You can envision a complex definition of popularity that's the pageviews but you could subtract off bounces and exits (or any of the fields).
sort: one of the metrics
filter: since I was only looking for the question's show action, I filtered out all other pages. The final code for us looks like:

def self.report_results(profile)
    report = Garb::Report.new(profile)
    report.metrics :pageviews
    report.dimensions :page_path
    report.sort :pageviews
    report.filters do
      contains(:page_path, 'questions')
      does_not_contain(:page_path, 'edit')
      does_not_contain(:page_path, 'new')
      does_not_match(:page_path, '/questions')
      does_not_match(:page_path, '/questions/')
    end
    report.results
end

You can also add date ranges on this... by default it follows the GA conventions of returning the last month's worth, which is what we wanted.

This function returns an array of Structs, with two properties in each struct: pageviews and page_path.

If you ask for a long report, you'll need to page the results. Refer to the garb documentation to see how to do this.

Saving/Updating the data

  def self.update_page_views(report_results)

report_results.each do |row|

      if /\/questions\/(\d+)/.match(row.page_path)

        q = Question.find_by_id(Regexp.last_match(1).to_i)

        q.update_attribute(:pageviews,row.pageviews.to_i) unless q.nil?

end

end

end

This logic can be tested using MockRow described above:

class MockRow < Struct.new(:pageviews, :page_path);end

...

Question.update_page_views [MockRow.new('50',"/questions/333")]

Getting this scheduled and run with the correct credentials is the last piece of the puzzle, which I'm not covering here; see cron, DelayedJob and your host.

Benefits

As you can see, there's nothing that tricky about this. A pair of us had this up and going in a couple hours (although we did spend time getting the DelayedJob running). There are a few important benefits:

It's a cleaner architectural with less server load than doing it yourself. You don't need to pollute fundamentally read-only operations with database writes.
Better metrics. GA can give you a more sophisticated metric than you could do easily. For example, it's easy to collect raw page views, but collecting unique page views or sessions is a bit more work.
You can change what metrics you use in an ad-hoc basis. For example, you can decide to only count posts from the last week in the most popular, and it's a simple code change. More interesting, you
Can eliminate or at least reduce developer and testers from metrics discussions. You won't have to be there to answer "can we make the most popular pages the ones that people spend the most time on?" If it's in GA, you can us it. If the product owner understands GA, she can figure out how to define "most popular" to produce the results she wants.
If you have already been running your site for a while, but are adding most popular support, you may already have a rich set of data on hand. This wouldn't be possible if you rolled your own.

I think this will be a core part of any new site I develop. It's so convenient to have access to this rich data set without any of the burden of collecting it. I hope it works out as well for you,

-- Andy

Other Versions
There are wordpress (drupal, etc.) plugins that do the same thing, but nothing that would work directly for us. For example, http://www.myoutsourcedbrain.com/2009/11/blogger-most-popular-posts-widget.html

Assert Changes and Fixture Test Helpers

2010-01-26T21:21:00.000-08:00

About a year ago I posted some test helpers for checking pre- and post-conditions during a test. I called them "assert_changes" and "assert_no_changes". They took a ruby expression to evaluate, a block, and did what you expected:

    o.answer = 'yes'
    assert_changes 'o.answer' => ['yes','no'] do
      o.answer = 'no'
    end

Since then, I have discovered similar functionality in shoulda and rspec. But if you're using test unit, and you don't want to take the leap to shoulda (which is painless), I packaged up my test helpers into a gem that's easy to install. Plus, mine are a little easier to read, if you don't mind evaling strings within your tests.

It's called "ayudante". To install,

    gem install ndp-ayudante --source=http://gems.github.com

Also included in the gem are "fixture helpers". These make it easy to test for sets and lists of your fixture model objects, without the hassle of building up your own lists and sets.

On goBalto, Ingar had created something like it for testing Sphinx search results. We had sets of fixtures, and we wanted to make sure a search returned certain results-- sometimes contains, sometimes exact. Sometimes we didn't care exactly what the order was so we wanted set comparisons. Or we didn't care about extra items.

Here's how it works. If you have a model object and fixtures for CandyBars, it adds test helper methods for assert_list_of_candy_bars (using #method_missing). Like all xUnit asserts, you pass the expected values (a list of symbols identifying the fixture objects), and the value you are testing.

    result = CandyBar.find(:all, ... )
    assert_list_of_candy_bars [:mars, :eminem], result

It also supports assert_set_of_candy_bars, so you can ignore the order of comparisons; and assert_contains_candy_bars so you can make sure it results contain a subset. Enjoy!

Recipe for 5 Whys with an XP Software Team

2010-01-26T20:15:00.000-08:00

5 Whys is a great way to get at the root of quality problems. On my last three projects, when I felt like code quality was dropping, I ran a "5 Whys" session. I have found it adds variety, solves a very specific problem, and plugs right in as an alternative to an agile reflection.

It's not in every agile software team's bag of tricks. Asking around our fairy savvy office, I discovered it's far from universal. In the "State of Agile" report from Version One, which includes survey results from 2500 software developers, it wasn't mentioned. Since I haven't seen it show up that much in other agile writings, I thought I'd share my experiences here.

What is "5 Whys"? I picked up "5 Whys" from the lean software movement, which sprang from Toyota manufacturing. You can read about its history on wikipedia, but it's pretty simple: at the end of the assembly line, when a widget comes out with a problem, you stop the line and ask "Why?" Whatever the reason, you ask the "Why?" again. Repeat at least 5 times. The goal is to discover the "root cause" of the defect, and fix the root cause, not just some symptom. Wikipedia has a good example around car repair. Here's a software example:

1. Why did Sheryl say the sign-up flow was broken?
A: Because she was trying to sign up a second time. Duh.
2. Yeah, but why did the flow break if the user is already signed up?
A: It's a bug. We didn't have a test case for that.
3. Why didn't we have a test case for it?
A: I just didn't think of it.
4. Why didn't you think about it?
A: The story seemed easy-- I guess I didn't think it through.
5. Why do you think you didn't think it through?"
A: I guess I was working alone and was in a hurry.
...

Obviously you don't have ask "Why?" exactly five times, but that does seem to be a pretty good number. The point is you start articulating the behaviors and procedures that cause the problems.

Taking It for a Spin. I've come up with exercise for agile software teams. Like I mentioned, when quality drops below your comfort level, give it a try:

(2 minutes) Make a list of the most recent bugs you've uncovered. I've found that the last 10 is usually enough-- if not too much. You can put these on a white board, index cards or a wiki page-- all will work.
(2 minutes) Prioritize them based on the ones you want to talk about. Which caused the most embarrassment or cost? You usually only have energy to talk about 5 or 6 of them.
(15 minutes) For each bug, ask "Why did this happen?" Then, probe deeper until you can't get much further. Write the answers down where everyone can see.
(5 minutes) When you're done with all the bugs (or out of time), circle common root causes.
(10 minutes) Brainstorm ways to mitigate or eliminate the root causes. Come of with one or two SMART goals for the team.

Some of the root causes we've gotten to recently:

we didn't pair on hard features
we didn't ask for clarifications on the requirements
one engineer didn't understand the intent behind the code
nobody looked at the app on Internet Explorer 6

Just like in agile reflections, the changes made coming out of these meetings stick with the team. During one session, the developers decided that there were just too many mistakes that someone else would have easily caught-- so they always needed two sets of eyes on every checkin. I'd advocated this before, but it was never applied consistently. To my surprise though, after the reflection, everyone was disciplined about this policy-- and our quality was much better.

Pairing with Designers

2010-01-09T19:59:00.000-08:00

I've worked on software with designers for 15 years, ever since software had a visual design. Usually this involves being "handed off" designs, or providing "feedback", via email. Only occasionally have I worked side-by-side to solve visual design and interaction problems.

Reflecting back, this seems sad, since working together has all the advantages of pair programming-- it's fun and educational, often much faster, and you can produce a superior result.

There are many blog posts about the merits of pair programming, but none about pairing between programmers with a designer. Since Carbon Five values pairing and collaboration so heavily, I've been trying to do it on all my projects. On my most recent project for the National Campaign, I've had the pleasure of "pairing" with several designers.

The first day of the project was refreshing-- I sat side by side with Jef and we broke apart his Illustrator designs, reassembled them in HTML, and fluidly passed ideas and png files back and forth. He was standing next to me, we were sharing a dropbox, and it was very exciting.

Five months in, a recent experience left Suzanna (our current designer) and me in awe of the merits of working closely together.

The task was a quick page redesign-- we have 17 detail pages utilizing the same basic template, each about a different contraception method (it's a site about birth control). From early user testing, we'd learned what wasn't working, and had reached consensus on the set of problems we should try to fix. Suzanna synthesized the feedback and produced a mock-up-- a single PNG screen mock-up representing all 17 pages. Pretty traditional so far. Among other things, the new design called for altering the main photo from its prominent placement in a rectangle taking up half the page, to a smaller photo that is elegantly wrapped by the initial paragraphs of text.

Traditionally, this doesn't go well: Some of the methods are quite small (the size of a match stick), whereas others, like the female condom, take up 6 inches or so (depending on how you measure). They have different shapes and visual weights, so it's going to be hard to get one design that works for all of them.

Suzanna specified a reasonable size for the image, taking this into account as much as possible. She develops a compromise based on lots of factors, including a guess about how a browser might wrap text. The developer would follow the spec as close as possible, and either wouldn't notice or say anything if there were visual problems.

Often the process stops there. You have a few pages that look good, and the rest various degrees of bad. Everybody argues that they did their job, and they did.

So the team tries to fix it. The designer might spec different dimensions to see if something else will work-- but this is simply another big compromise. Or the designer will embark on the tedious task of creating a mockup for each page-- making them beautiful with Photoshop's font rendering and text wrapping (which the developer can't really duplicate). In the best case, a compromise will be reached that looks just okay for all the pages.

Things went differently, because the programmer and designer were sitting next to each other:

We are working with a 960 grid, and Suzanna made everything fit in our grid. As I started the implementation, she measured and confirmed. But fitting in a grid for a designer can be different than fitting in a grid for a developer.

I probed, "What about that image-- it's so tall and narrow. You want that to stick up above?"

"Naw... Well, maybe. Why don't we just build the images with 20px "cushion" that the image can bleed into, in case they need to extend?"

"Sure," I said. I'll use negative margins to break out of the grid (a little):

#photo {
float: left;
height: 180px; width: 180px;
margin: 0 0 -5px -20px;
}

Even if we'd stopped there, our implementation would be an improvement over a traditional outcome without the negative margin bleed. Suzanna could now have shadows extend into the margins, which will be a nice visual effect.

But that's not the end of the story. I walked back to my desk and implemented the page while Suzanna cranked out the images.

She interrupted me five minutes later, "This one's taller, is that okay?"
I was in the midst of placing the image and realized I could add one CSS rule that would handle this. "Sure!"

#photo.the_pill { height: 241px; margin-bottom: -20px; }

Ten minutes later Suzanna came over to my desk, "I finished the images and they're in your drop box. I had to make a couple other ones different sizes."

She'd named them consistently with the old names, so I just copied the files over. We compared my implementation with Suzanna's PNG mockup. Close, but not close enough. Suzanna sat down next to me and we pushed a few pixels around, getting the margins right with the browser's font metrics-- the line heights were slightly different she needed an adjustment. A quick check on Windows as well, and it was perfect.

Then we looked at the odd size images she'd discovered, and we tweaked those sizes and margins as well. Some of them needed a little margin adjustment as well. No problem-- we just flipped back and forth between browsers and CSS files making everything perfect. (She didn't actually need to make any measurements as we just used our eyes.)

While we were at it, we ran through the all 17 pages, just as a quick check. The set of non-exceptional pages should be fine, but Suzanna wanted more adjustments. Each photo is unique. Since it was so easy, why not? In a matter of minutes, half the pages had some custom pixel adjustments, and we were quite happy with all of them.

#photo.diaphragm { height: 195px; margin-bottom: -20px; }
#photo.withdrawal { height: 201px; margin-bottom: -40px;}
#photo.depo_provera { height: 189px; margin-bottom: -20px; }
#photo.iud { margin-top: 5px; height: 226px; margin-bottom: -60px; }
#photo.the_ring { margin-bottom: -20px; }
#photo.cervical_cap { width: 195px; height: 192px; margin-right: -15px; margin-bottom: -20px; }

This probably took less than an hour. This isn't very complicated, but it's hard to imagine how tedious this would have been if we hadn't paired to reach the final result. Plus, each page really looked great-- the image tucked in "just right."

I've certainly collaborated with designers before, poking around in Firebug until things looked good. But this was the first time I saw how clearly working closely produced a superior result. And we did it with no tedious measuring of text or dozens of time-consuming mockups.

What could have taken hours or days took minutes. We just sat down together, and, well, paired.

Scrum, XP, Agile, and Visualizing the Difference

2009-09-24T21:50:00.000-07:00

Visualization of agile practices running here:

http://ndpsoftware.com/agile_methods/agile_methods.html (If it seems to draw garbage all over the screen, refresh the page. More on that later.)

What it shows is the main concepts of 5 different agile viewpoints: XP, Scrum, the Agile Manifesto, Lean and Getting Real. Important words for a practice are "attracted" to (gravitationally) to the practices that mention them in their canonical definition.

Motivation

There are different flavors of agile. I discuss the differences coherently, but I wondered if there were some way to distill the differences down to something that could be represented in one page. A cheat sheet. All these practices share some history and concepts, but can be they be presented and compared in a fairly simplistic way? Each emphasizes different things, but how do they compare?

Building the Visualization

The first challenge was to identify the top few values and concepts that define each practice. My idea was to summarize each practice by finding the canonical description of each practice and capture the top concepts as succinctly as possible.

So I identified a primary source for each practice. This is arbitrary but not capricious. I believe I was picking reasonable documents:

XP: http://www.extremeprogramming.org/
Agile Manifesto: http://agilemanifesto.org/, http://agilemanifesto.org/principles.html
Scrum: Agile Software Development with Scrum, Chapter 9; and http://en.wikipedia.org/wiki/Scrum_(development)
Lean Software: http://en.wikipedia.org/wiki/Lean_software_development
Getting Real: from page 2 of the book

Next, I needed to extract the concepts. I created a spreadsheet and listed concepts down the left side, and practices across the top. I put a number in any box where the practice mentioned the concept as one of its first. The first concept mentioned on the page got a "1", and the 10th one got a "10". I didn't collect concepts much beyond that. I ended up with a big grid with practices as columns and concepts as rows. As the terminologies of each practice differ, I had a sparse spreadsheet with only a few data points. It wasn't helpful in comparing the practices.

So went through a normalization process. Is a "sprint" the same thing as an "iteration"? Well yes (and no). What's a "project heartbeat?" and is it the same thing? I combined rather aggressively the first time I did it, but what you see is a second, more conservative approach. There are still problems, but I took a pretty good shot.

Even with the "normalized spreadsheet", though, it wasn't easy to compare the practices. I tried different sorting and coloring to no avail. I then tried different graphing options, starting first with "mind maps". I tried 3-d bar charts, but they were little help. And plotting the words in grids, but couldn't make it work.

I finally stumbled upon the visual thesaurus metaphor. As a regular thesaurus, I always thought the display a bit gratuitous: a word in the center with its synonyms "floating around", connected by some gravity. The more I thought about it, though, the better fit it seemed.

Next I needed a tool in which to implement my visualization. You can license the technology, but I had a $0 budget and just a few hours to devote to it. I looked at using Processing (a Java variant that lots of people are using to build visualizations); there were some implementations, but I wanted something even lighter weight if possible. After another Google or two, and I found Javascript-based experiments with "force directed graphs". I grabbed the code, and threw my data in it.

Actually, there was a little engineering involved. I knew I would update the spreadsheet (and I wanted to add other practices potentially), so instead of converting the spreadsheet directly to the Javascript, I wrote a Ruby script that takes the spreadsheet and generates JSON data structures. The structures are then fed into a small engine I wrote, and the graphic is created.

Code available http://github.com/ndp/agile_methods

Agile Software Methodologies Visualization by Andrew J. Peterson is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.
Based on a work at www.kylescholz.com.

Tips on Taking Autotest for a spin on OS X

2009-08-13T23:39:00.000-07:00

It's exciting to get autotest working for my last rails project. In case you don't know, autotest is a tool that continuously runs your ruby/rails tests as you work. It monitors your file system watching for changes, and then has a heuristic to figure out which tests to run. Once a test fails, it keeps re-running it until it passes.

Gotta say, the jury's still out, but overall I really like it. Lots less navigating around tests and hitting shift-ctrl-F9 (or whatever it is). I got inspired after listening to a year-old RailsConf talk by Kent Beck about the origin of TDD (after discovering we went to school together at UO).

There are other tutorials that walk through the installation, so I won't repeat them here. But the additional information you need is distributed in 3 or 4 posts across the net, so here they are collected:

The first trick is to get a couple additional gems (besides ZenTest, which is what you need for autotest).

You will also want:

sudo gem install autotest-rails redgreen autotest-fsevent autotest-growl

Autotest-rails teaches autotest how where tests are in Rails projects (as if there are any other kind). Without you won't see any tests running. RedGreen adds nice coloring. FSEvent makes it faster on MacOS X, and autotest-growl, well, growls.

Second, I didn't find a complete listing of what your ~/.autotest should look like on OS X. Mine looks like:

require 'redgreen/autotest'
require 'autotest/growl'
require 'autotest/fsevent'

That's it! Definitely give it a shot. It's most convenient if you have another screen for the autotest (IDE integration would be nice). And an extra CPU wouldn't hurt, because it keeps my MBP CPU pegged much of the time.

Working with Agile Developers

2008-02-15T20:37:00.001-08:00

I recently joined Carbon Five, a small agile consulting company specializing in boot-strapping start-ups. I got a chance to work with C5 over the last year (as a client), and I learned that the expectations of working with us are very different than working with other consulting companies.

What is agile

Agile is a software development process emphasizing:

close collaboration between the programmer team and business experts, or in our case, between the client and developers

face-to-face communication (as more efficient than written documentation)

frequent delivery of new deployable business value

ways to craft the code and the team such that changing requirements is not a crisis.

Agile teams strive to collaborate in-person as much as possible. And we deliver software not in just a few key milestones, but continuously: weekly, daily or hourly.

What to expect from us

Working with us is collaborative and iterative. You tell us what you want, we listen, and together we identify the most important features. The goal is to identify the features (or "stories") that provide the most value to you and your customers. Once we agree, we will build it quickly-- not in months or years, but weeks.

We will deliver this early version so you can try it out. Once you experience it, you will likely have a reaction-- a good one, we hope. But if you don't like what you see, that's fine. We haven't invested much, and we are early in the process. We can easily adjust what we're building. It may inspire new ideas or a change in direction. Whatever the reaction, we're not locked in. Hence the term "agile".

We meet again, and prepare to repeat the cycle. Again, we'll address the highest-value aspects of the project, deliver them, and we get feedback for the next iteration.

What not to expect: we are different than an "outsourcing" experience. In that engagement, you expect to tell the agency what you want, and they deliver it to you in three or six or 12 months. (Easy? Yes. But it seldom works easily.) We expect you to be engaged in the development process, defining requirements, setting priorities, and providing feedback.

Why agile

Agile, and the related methodologies like scrum, extreme programming, lean/agile, is a response to the failures of "traditional" software development. Even though the "traditional" process has been around for nearly 40 years, it has never worked particularly well.

64% of features included in projects are rarely or never used

The average project exceeds its schedule by 100% and nearly two-thirds of projects significantly overrun their cost estimates

Agile, with its different emphasis, addresses these:

The most common reasons for software failure are unrealistic or unarticulated project goals and badly defined requirements. We build working software in collaboration with our customers. By collaborating, we identify missing requirements right away (and resolve them with you.) And our customers see problems right away, and get a chance to respond to them. This tends to reveal right away when our goals aren't aligned.

The other common problem is communication among customers, developers and users, and poor reporting of the project status. Agile addresses these head-on. At it's core, it's about communication. The team meets daily to identify problems, programmers work together ("pair"), and the customers are involved frequently during the development of features. In addition, software is delivered and reviewed frequently (usually every two weeks), so the team-- even if there is some huge misunderstanding-- rights the ship before it can get too far off course.

What we expect from you

Collaborate with us. We expect to be collectively figuring out, and then building, what you want.

Here's a typical cycle: We will periodically (every couple weeks), have a planning meeting. Here, we talk about what you want: this will be business goals, product features, and perhaps "the color of that button over there." Once all the ideas are on the table, we will estimate the amount of effort they will take to get done. These are estimates, but help you prioritize the work: a medium priority feature may be worth doing if it's really easy, but should be shelfed if it is a huge undertaking. We keep the developers involved so you can make informed decisions. We need to you to be actively engaged in these meetings and strongly representing your needs. The outcome of these meetings largely shapes what we will do over the next iteration.

We will expect a daily engagement. We will need you, or your delegate, to answer product questions throughout the business cycle. By foregoing huge, unmanageable requirements documents and product specifications, we have saved weeks or months of time. But when we do run into a question, like (insert example here), we need the ability to get an answer fairly quickly so that we can move forward.

Finally, we expect you to make time to look at the working software we deliver and respond to it-- not wait three months until the project is done. Expect from us working software.

Notes:

http://www.agilemanifesto.org/

http://www.agilealliance.org/show/2

http://www.it-cortex.com/Stat_Failure_Rate.htm This article aggregates several studies on software failure rates.

http://spectrum.ieee.org/sep05/1685 Why Software Fails, Rober N. Charette, September 2005. Summarizes why projects fail.

Unit testing time

2007-11-01T22:41:00.000-07:00

We've all run across the problem of trying to unit test some code that is dependent on the current date. Such as:


   public String elapsedTime(Date d) {
      final Date now = new Date();
      final long ms = now.getTime() - d.getTime();
      return "" + ms / 1000 + " seconds ago";
  }

How do you test something like this? (or something more sophisticated and useful)

I found the solution in a posting about mock objects. The trouble is that this code is depending on the Date class, and its behavior of getting the current time. Imagine a Clock interface:


  interface Clock {
      Date getNow();
  }

and then a rewritten elapsed time:


  public String elapsedTime(Date d, Clock clock) {
      final Date now = clock.getNow();
      final long ms = now.getTime() - d.getTime();
      return "" + ms / 1000 + " seconds ago";
  }

Obviously the Clock can be injected in other ways, but now we have a testable and extensible method.

Scriptaculous Effects Demo

2006-12-17T23:00:00.000-08:00

I threw together a demo of Scriptaculous effects: http://ndpsoftware.com/ScriptaculousEffectsDemo.php Nothing crazy here, but I'd like a flexible and interactive demo. Send me ideas or suggestions, Andy

Powerbook Battery Rejuve

2006-05-06T23:04:00.000-07:00

My aging G4 Powerbook battery slowly died over the last month or so. It just seemed to hold a charge less and less. Last week it wouldn't go on for more than a minute or two. This weekend, unplugging the power cord was equivalent to a shutdown. After that, the battery reported "missing". It seemed like a strange progression-- not as organic, following some sort of decay curve, as I would have expected the material to behave. So I looked on the Internet to see if there were any clues. There were a set of my batteries that were recalled, but (damn), mine didn't qualify. Next, I found a thread about resetting p-ram and other such shenanigans. I was skeptical, but it seemed like there was one trick that worked for people: going into open firmware (Restart with Cmd-option-O-F pressed), and then resetting the chip that manages the battery power (reset-nvram). It was worth a try. So I did this. Wow, what a transformation. Now my battery seems to hold a charge just like it did a month or two ago. I haen't gone through extensive tests, but it's workable again. Perhaps that other dead battery will come to life... -- report from a year later -- Battery life wasn't that great for very long. It decayed to be mostly unworkable after a month or two. After a year it was down to ten minutes or so, but never as bad as before. Yeah, I've bought a new battery.

Page layout options

2006-04-09T23:06:00.000-07:00

I put together this little experiment to show the options and technique for doing different page alignment strategies: fixed width, stretch to the browser, or various hybrids you see around the web.

My question was,

can I just wrap a site in a div or two and then decide this later?

YES!

http://ndpsoftware.com/experiment/css/stretch.php

First grade first

2006-01-06T23:02:00.000-08:00

Is using temporary variables was messy? I now advocate writing 1st grade code. By that I mean "See Dick run. See Jane smile." etc. Hey, we're all adults here, you say, why can't I write real code.

Well, writing really simple code makes debugging easier. What do I mean? I've had a couple occasions in the last week where debugging would have been easier if I'd writing really simple code. For example, I just got an null pointer exception here:


          _dataSetsNeedingCleanup.add(new DataSetRef(state, dataSet.getId()));

I can't easily tell where the problem is coming from. It could be one or two places: my member variable _dataSetsNeedingCleanup could be null, or dataSet itself code be null. If I had written it like this,


          final DataSetRef dataSetRef = new DataSetRef(state, dataSet.getId());
         _dataSetsNeedingCleanup.add(dataSetRef);

a stack trace provided by the user tells me exactly what's wrong. This saves me a whole step in the debugging cycle!

2005-04-01T23:08:00.000-08:00

We posted on craigslist.org for a jr. java engineer and gotten the predictable hundreds of responses. I have seen all sorts of mistakes. Normally I wouldn't repeat this sort of information in a public forum, but what the heck, it's a blog.
COVER LETTER
Follow the directions. If it asks for a cover letter, send one. I kinda like this one: "Hi, Here is my resume. Thanks", but most people would find it too brief. Really long ones show you can't prioritize your work. Generic ones that don't match the listing generally disqualify you. If you're going to try to make the point that the job sounds "interesting", you MUST explain why. Nobody's done it yet. After reading it 100 times, "please email or call me at anytime" seems redundant.
GENERAL
Don't hide behind words. I'm only going to read 20 or so words when I skim your resume. The fewer words you have, the more likely I am to see the important ones. Have someone proofread. I'm waiting for a well-written letter. I don't care if you worked at the cineplex or best buy. Don't attempt more formatting than you can handle. Understand the limitations of MS Words or email before going wild with formatting buttons. It looks unprofessional if certain words or phrases are different sizes. Be aware that not everyone will have the same fonts and OS as you. One person put their resume into pdf, but somehow managed to completely mangle it.
SUBTLETIES
Don't cc lots of companies. Think about the name of your resume. "oldresume.doc" is curious. WORST TRANSLATION
"References available on demand."
BEST FOLLOW-UP
"...Hey, good luck with that. Ya know, you'll probably find someone too, market being what it is. Sucks."