Charles Roth       (Techblog top)
Started 8/25/2009.
First released draft: 9/18/2009.

I. Introduction
I began work as a Pillar employee on 6/8/2009, on-site at Gale Research, jumping into the middle of the Career Transitions ("CT") project.  The work has been fun and educational, and I hope to continue with it, if and when CT resumes.

But!  I also saw individual and group behaviours that intrigued me: good and bad, intentional and unintentional, explicit and implicit.  The time spent at Gale was a mother-lode of gold nuggets for the would-be "Agile anthropologist".  I'm no Margaret Mead, but I do have a life-long interest in group dynamics and anthropology... and unlike most software developers, I also like to write. 

Thus, this article. 

There were many implicit (unexplained, undiscussed) behaviours of the team that surprised and sometimes concerned me.  Some of these may be worth notice and possible discussion inside Pillar.  To summarize, my purposes in writing this article include:

1. Point out implicit "fish-in-water" team behaviours that concerned me, on the chance that they might be something that Pillar could improve on.
2. Point out the same to Pillar folks who might be interested in discussing them... because I might learn more that way.
3. Scratch my itch, in good Agile/open-source style.  I can't believe <grin/> that no-one has written about these things: so it might as well be me.

Just to emphasize... this is not a complaint or a diatribe.  Where I name names, it is only to point out what I consider exceptionally positive behaviour.  I mean for this to be feedback in it's truest, engineering, sense. 

II. Context
Just a little bit about me and where I come from, to provide context to my observations.

• Wrote my first Fortran program in 1970 at the age of 14; computers hooked me and I've never looked back. 
• Graduated B.S. Comp Sci from Rensselaer Polytechnic Institute in 1977. 
• Taught computer science 1980-1982 at Wayne State University.  Loved teaching, but the pay sucks.
• Lived through, embraced, and helped teach many paradigm changes: structured programming, C, OO, the Web, founding/surviving a $3M .com/.bomb, and now Agile/XP.  Confident enough to believe I can help others embrace new(er) practices, humble enough to remember that my thinking will get up-ended at each new shift.
• I am new, and not new, to Agile.  I spent three weeks at Menlo Innovations in May 2006.  Much of what I learned about XP there had the "aha" shock of "this is what I've been fumbling towards for years!".  But I found Menlo too rigid, too cult-like, to stay.  I then tried to drag a small U.S. SaaS company (with mostly remote Chinese developers) into Agile practices, teaching myself simultaneously.  I made a lot of useful changes, but it wasn't pretty, and they never really embraced TDD.
• So I explicitly chose to move on to some place that really believed in Agile, but was not rigid.  Hence, Pillar.

III. Field Notes
These are literally field notes about the behaviours I saw that were interesting, or that concerned me.  I have organized them into categories that (roughly!) parallel published Agile methodologies.

I have tried to clearly separate out my recommendations (in separate blocks following each section), although I confess to some (gasp!) interpretation of each behaviour. 

1. Pairing. 
   1. Pairing newbies.  During my first week at Gale, I was paired with another new person, Laurence Rama.  This seems counter to normal pairing practice... but oddly enough, it worked very well.  We were given cards about new features that were reasonably transparent to anyone familiar with any java/web platform, so we were able to add some velocity immediately. 

      Because we were both new, neither of us deferred to the other, and we were able to make good use of our complementary skill-sets.  This was either brilliant planning on Matt Van Vleet's part, or damned good luck, as there were other possible first pairings that would not have worked nearly as well.

      Recommendations:  When this happens, assign a specific person -- not the team manager, not the tech lead -- to be their immediate go-to person for questions.  It's good when a newbie pair can work things out together, but some may hesitate to ask questions when they're stuck.  (Fortunately, I have no fear of looking stupid, so I asked most of the questions for our pair.)

   2. Vague pairing rules.  During my initial interview, I was told that pairing was frequent but often optional.  During my first few weeks at Gale, almost all work was paired.  But as time went on, and as some people were slowly moved to other projects, pairing behaviour changed a lot.  It was frequently unclear whether we should pair, with whom, and why.

      At one point, my then-pair "P" and I were working on three cards A, B, and C:  P on A, me on B, and "we" on C.  All three cards had significant built-in delays in the test/develop cycle, which is what led to this curious arrangement.  We did examine each other's work on A and B before check-in, to at least preserve the spirit and some of the benefits of pairing.  It was efficient, it worked, but it was weird.

      Even later, when the number of developers grew very small, I usually worked solo, but grabbed another person any time I had a doubt, felt like I needed a sanity check, or before check-in.  This worked well (as far as I can tell!), but it was never really discussed explicitly.

      I did at times return to traditional pairing when there was a need for deep knowledge sharing, or for me to pick up another's work that was in-progress while the first person went on vacation.

      Recommendations:  Be flexible about pairing, but be explicit about the flexibility.  Don't force pairing for everything.  Don't force pairing unless you really mean it.  Otherwise it lowers morale and encourages cynicism.

   3. Pairing difficulties.  The most fascinating thing about Agile, from an anthropologist's point of view, is pairing behaviour.  With essentially no instruction or explicit norms, most of the developers seemed to pair readily and effectively... even as the styles of pairing varied widely.

      There were a few difficulties that I saw on repeated occasions that are worth noting:

      • Large disparity in project-specific knowledge.  I've been on both sides of this pair. 

        In one case, I was new to the team, and simply overwhelmed with the level of class dependency knowledge needed in order to work with my (extremely thorough and knowledgeable) pair partner.  After several days, I recommended that I was not ready for this pairing, that it would take too long for me to come up to speed, and that I was slowing my pair down.  He agreed and was very gracious, and I moved to a different pairing that had worked on cards with smaller scope, which gave me a chance to absorb more about the class structure without being overwhelmed.

        In a different case, near the end of my ten weeks, I paired with a young newcomer, recently out of college.  I specifically picked part of a card that I knew she could handle, tried to give her a "10,000 feet down to this point" overview, and handed her the keyboard.  Success was mixed, but she was able to be productive.

      • "Introvert thinking".  There is a tendency for introverts to think silently, to invisibly go from A to M, and then surface -- which can leave everyone around them wondering how the heck we just got to N?  This can be quite confusing in a pair, where frequent conversation seems to "grease the wheels".  This is especially true if the introvert is a senior team member, and the partner is new and more hesitant to insist on step-by-step communication.
      • High bandwidth, low communication.  The reverse of the introvert case is the pair partner who talks almost constantly.  This "high bandwidth" can reduce actual communication, and stir uncomfortable feelings in the other partner... who may end up feeling responsible for managing both sides of the conversation.  The high bandwidth partner is frequently very skilled, knowledgeable, and productive... but the situation can require great strength and delicacy from the other partner.
      • Ambiguous speech: demonstrative pronouns and adjectives.  Words like "it", "that", and "this" in pair conversation about classes and objects are prone to ambiguity, especially in rapid or high-bandwidth monologues. 
      • Shoulds vs. Wants.  Different members of a pair naturally approach any given problem (e.g. defining the responsibilities of a new class, or what a reasonable set of tests would be) with different assumptions.  On multiple occasions I observed the beginnings of conflicts about just "how far" the responsibility of a class should go.

        What I found helpful in these cases was to be explicit about what we each thought a class should do at a minimum, versus what we each wanted the class to do.  (In one case I actually drew out a 2x2 matrix of "should do/not-do" vs. "want to do/not-do" to make it clear.)

        The danger here is when pair partner A interprets B's "wants" as "shoulds", and begins an argument about it.  When the distinction is made clear, the "virtual argument" disappears and forward velocity resumes.

      • Dunning-Kruger effect.  A fancy name for a common phenomenon.  In rare cases, a pair member may be so utterly confident -- and wrong -- about something, that the other partner may hesitate to question them.  In one (and only one) case in a triple, I felt that I had to be rude in order to demonstrate that we had gone down a very wrong track.
2. TDD.  More to follow in a later draft.
3. Story Cards. 
   1. The quality and legibility of story cards varied enormously.  Most of the time it was necessary to find the person who wrote the card, and get them to explain what they meant.
   2. Cards were taped to the (former) cube-walls.  They frequently fell off, and were retaped, and fell off...
   3. Mark Price repeatedly had to ask people to write in their actual times on the cards, once the card was complete.  There were only vague (and frequently differing) understandings of how to record time on a card.

   Recommendations: 

   1. Provide a few made-up sample cards, that show "best practices" for writing cards.
   2. Require card writers to put their name and a title in big letters at the top of a card.  Cards without those get moved to a visible "not accepted" board.
   3. Allow developers to take a really ambiguous or unclear card back to the card-writer, and require the writer to rewrite the card.
   4. Write short examples of how to record and fill-in times for cards.  Put them in the wiki.
   5. Find a better way to make cards stick.  I cut a 4x4 foot piece of homasote, screwed some L-brackets onto it, and hung it over the cube walls.  People could tack or tape cards to it with having them fall off.  I'm sure there are even better/easier ways.  (Maybe duct-tape a big piece of stiff laminate to the wall, and tape cards to that.)

4. Information Sharing 
   1. Word of mouth.  Works great most of the time.  Sometimes it's hard to know when to interrupt others; sometimes we get layers on layers of interrupts happening.

      Recommendations:  This belongs in a retrospective.  Each group is different.  How should we interrupt each other?  5 minutes of explicit conversation about it in the group might save several person-hours of hesitation time.

   2. Written information.  This group seemed positively hostile to writing anything down.  Some key pieces of information got passed around via email, but we never even had a consistent/updated email list(!!).

      There is an understandable Agile reluctance to commit anything to writing that isn't code.  That makes sense, far too much documentation gets out of synch with live code 2 minutes after it is written.

      But I suggest that this can go far beyond the natural Agile mindset; it can instead become a rationalization for avoidance.  When you have to tell more than 3 people something, it's time to write it down -- even if you know it will become out of date.  Especially if you know new people will be added to the team later on.

      Recommendations: 

      • Build a team email list -- a real list with an alias, not an email with a bunch of cc's that gets passed around.  The latter is a guarantee for failure.  If you could write a test for it, you know it would fail.
      • Build a team wiki.  Use it!  Commit the little nuggets of key information to it.  (A handful of real examples: "How do I turn on/off eclipse/liferay debugging".  Or "Where did we put the modified Liferay files we have to plug-in by hand?".  Or "How does LinkedIn integration work?"  Or "What's the URL to access the integration server report from my desktop?"  Or... you get the idea.) 

        If you're tempted to send an email, don't -- put it in the wiki, then send the email with a pointer to the relevant wiki page.

      • The "tech lead as benevolent despot" role works well, to cut thru uncertainty.  So appoint a benevolent despot to rule the list and the wiki.  (It should not be the tech lead.)  Give him/her the power to insist that people use the wiki; if developers try to use email, the despot gets to bounce it back and tell them to immediately put it in the wiki.  Make sure the despot gets a little time now and then to keep the list and the wiki clean.
      • Don't depend on "someone" at Pillar to maintain blogs or wikis.  Make it local, control it locally.  (OK, maybe it can live on a Pillar server.  But it's got to feel local.)  One senior Pillar staff member told me "The Pillar Wiki is where information goes to die."

   3. Classes, objects, naming.  The book "Clean Code" talks about readable code as a domain-specific language.  That means that class names have to make sense in the context of a larger language.

      Recommendations: 

      • Define some domain-specific key-words at the outset, or along the way as concepts evolve.  Write them on a whiteboard, a wiki, a shared file, anything.
        OR:
      • Write 2-3 sentences at the top of each class, describing what the heck this thing is.  Not what it does, not how it works, but what it is!  Or even why it is!

      Granted, both of these are edgey -- they slide towards waterfall, towards documentation rot.  But you cannot have a language without a basic vocabulary, and you cannot have readable code in the style of "Clean Code" if it doesn't at least hint at a domain language.  Otherwise your "WTF?" count increases, as you end up with not-very-meaningful class names like UrlTarget and method calls like BigBlob.getTextSize().

5. Integration Server. 
   1. The integration server was mysterious.  There was a fair amount of knowledge about its operation, and the norms for using, that just didn't live anywhere, and couldn't easily be described by word-of-mouth.  A very few people worked hard to keep it clean, and it was an honorable but unenviable task.
   2. The integration server builds broke.  A lot.  If anything, the longer the project went on, the more often the builds broke.  There was much good-natured pleading (esp by the tech lead) for people to keep the builds from breaking, or to clean up after a build broke... but there was no obvious pattern that anyone was doing anything in particular wrong.  It was also hard to tell who was really responsible for any given build breaking.

      Possible reasons for the frequent build breaks included:

      1. Multi-phase commits.  Builds commenced on a commit, but since the production code and the integration tests were in separate projects, a commit to one could trigger a breaking build, because the integration test project commit had not yet happened.  "False positives" like this encouraged a "just wait 'till the next commit" attitude.
      2. EasyB test failures.  These were often the "shoemaker's children" of the project, possibly because there was confusion about who was writing them from the beginning.
      3. Database changes.  Early on, all database changes were basically emailed around, which caused confusion and sometimes broke a build.  Eventually a subversion-based tool was added that automatically ran database updates, which was a vast improvement and deeply appreciated by all.

      And sometimes we just didn't know "why the damned thing broke" and just kicked it again and waited for (or even forced) the next commit.

   Recommendations: 

   • Reduce false positives.  Arrange projects so that developers can easily choose to run unit-tests separate from integration tests, yet both are handled by a single commit.  Or make a build commence on a commit, but only after a certain delay.  Or... I don't know, but this problem must have been solved somewhere already.
   • Make it easier for anyone to take responsibility for integration server cleanups.  A one-or-two page guide might suffice.
   • When frequent pleas don't get results, then lean on it in the retrospectives.  These are good people, if something's not happening, then the situation itself may be at fault.

6. Information flow / teaching / learning. 
   1. Lunch-and-learns.  Overall, these were great opportunities.  The publicity for them was mixed and often unclear.  If in doubt, I went anyway, and then left if it didn't seem useful.

      The best was Dan Wiebe's talk on EasyMock -- not just because I personally found it useful, but because he was very clear about what this was, why he thought it useful, and only then delved into details.

      The worst was a departing Gale employee talking about "ATDD".  I stayed for 45 minutes, and never actually found out what it was or why it might be useful.  (And as a former teacher, I was not, shall we say, a passive listener.)

      Recommendations:  As silly as it may seem, I think it would be possible to do a lunch-and-learn on how to do lunch-and-learns.  Subject matter experts do not automatically know how to teach, but there are some basic principles that can help.  The simplest is: start with who I am, what I'm talking about, and why that might be useful to you.  Don't dive into details right away, start at 10,000 feet and work down.  Do give short, specific examples of how something was useful.

   2. Books.  I found it highly ironic that, at Gale of all places, nobody seemed to use books.  There seemed to be a belief that, if you needed to learn something, you should find it on the 'net.

      A few people did pass around one or two books: if it looked useful, I immediately bought my own copy.  (jQuery in Action was particularly useful.)  I brought in a load of ~15 books that I thought helpful; a few people used them on occasion.

      There may be some tension here between "travel light" and "gain mastery" that I do not understand.  It troubles me, but I do not (yet) have recommendations.

7. Technology choices. 
   Obviously, the choice of technologies affects how well a project progresses.  But more subtly, the choice of technologies affects how well specific Agile practices work... or don't.
   1. The choice of Liferay as the base platform is a good example.  On the surface, it seemed like a reasonable choice.  But because of the way it "takes over" certain Tomcat functions, it had the affect of really slowing down the develop / test / deploy / test cycle.  Unlike "vanilla" Tomcat or JBoss deployments, with Liferay in the middle, it was impossible to just change a JSP and immediately see the results.  This wasted a lot of time, and caused developers to "batch" many changes together.

      Recommendations:  Liferay seemed to offer some features that would be useful: built-in user creation/authentication, and the handling and arrangement of portlets.  But in practice, neither of these appeared to help very much, and in fact required some serious "working around" their features at times.

      In 20-20 hindsight, this may be a case of "use the simplest possible tool for the problem".  I believe the overall velocity would have been significantly higher if we had built our own pieces just on top of Tomcat.

   2. The use of EasyB was a well-intentioned attempt to have business stake-holders write application/behaviour test skeletons, which would then be filled-in with Groovy code by developers.

      Unfortunately, it never happened that way, so the responsibility for the application tests was always unclear, and thus they always lagged behind "real" code.  And many UI problems had to be discovered by live human testing.

      Recommendations:  In this particular instance, we might have been much faster, and the responsibility much clearer, if the developers were told to just write the application tests directly in Selenium (no EasyB, no Groovy), and run them straight from JUnit.

      In general, the responsibility for the application tests should be made explicitly clear at the outset, and then let that drive the choice of technology.

IV. Closing Observations
Compared to my experiences in other similarly-sized environments, at Pillar/Gale I observed very robust code, moderately high velocity, and high developer enthusiasm.  I attribute this to these practices:

1. Buy-in of TDD across all levels.
2. Story cards (mostly) ensured good-sized tasks.
3. Developers could write story cards, especially for bugs, problems, and doubts.
4. The "shared honesty" enforced by pair-programming.
5. Frequent rotation of pairing.
6. Shared group norms of "Travel light", "don't speculate", and "simplest solution that works".
7. Strong encouragement of immediate refactoring.
8. Interviewing process that (mostly) ensures basic competence with all of the above.

But you knew that already, right?

What I found particularly interesting, however, was a group sense that somehow these practices could (or should) solve all the problems that the developers faced.  When these practices did not lead to solutions, I observed frequent pleas and exhortations to "do it better", "take more responsibility", "remember to fill in times on the cards", and so on.  The pleading was generally good-natured, but had a background feeling of helplessness, of puzzled frustration that "damn it, we're good Agilists, why aren't we solving these problems?".

Now, in many cases, some of these problems were solved, often by someone "scratching their itch" after an issue was repeatedly brought up during the retrospectives.  (The management of incremental database changes is a good example of a problem that was -- eventually -- solved.)

But my main point here is that there can be a disparity between what the team believes that "Agile can do", and what the situation actually allows.  And like any group with a strong shared belief system, when the beliefs fail to change reality, sometimes the frustration turns inwards to the team, that somehow "we were not good enough, did not practice our beliefs strongly enough". 

(I am reminded -- distantly -- of many "millennialist" and Utopian movements in the U.S. in the 1800's.  When the promised Second Coming or social revolution did not occur, these groups felt puzzled and defeated, and lost a great deal of their energy and focus.)

This "turning inwards" is actually much more positive than the historical response of frustrated developers -- to blame the management, choice of technology tools, lousy specifications, etc.  An inwards look can encourage developers to reflect on what they could do differently, what itch could be scratched earlier, and what different technology choices might help.

The retrospectives are a powerful tool to use this "turning inwards" and redirect the energy towards identifying the real problems.  Curiously, I observed, at best, "mixed" energy levels during the retrospectives.  I'm not sure why this was, but it was decidedly lower than the energy during stand-up meetings, or during daily development.  Possibilities include:

• Many developers just wanted to get back to programming.
• Pointing out the same problems repeatedly was discouraging.
• Problems would be assigned to some people (esp managers), but progress was unclear (not testable!)

V. Conclusion
I break with anthropological tradition here to (briefly) sermonize, and end with a question.

Agile/TDD turns the most common programmer vice -- "I just want to write code" -- into a virtue.  In doing so, it also releases a huge amount of energy -- the unspoken, often repressed anxiety that "my code will break the next time it is changed".  (Which can lead to "nobody but me should ever change it.")

Agile solves huge problems.  But having done that, there is a temptation to believe it solves all problems.  We must not, in our enthusiasm, fall into that trap. 

"Always test it!" is a powerful message and tool.  How can we extend it to problems that we have not been solving?