Five Books I’m Reading

There’s this meme that goes around the internet periodically that talks about how how most people never read a book after college.  Every time I see it, I am shocked and decide it can’t be true.  Or that the research that decided this had some sort of weird definition of book:  Must be dead tree, fiction, and by an author of particular nationality or note or something.  Because I can’t imagine ever *not* reading books.  And thanks to my e-reader (I use a kindle – use what you want) , I’m always in the middle of a bunch of books, sometimes rereading, sometimes gobbling up for the first time.

I’ve read that that Amazon does analysis of Kindle readers, determining how many people make it all the way to the end of each book.  And while I have that problem sometimes, most often, if I start something, I’ll finish it.  And probably reread it a couple of times too….

So what books are top of the carousel on my Kindle right now?

  1. The Meditations of Marcus Aurelius:   I almost wish I had a study group for this one.  After reading Ryan Holiday’s Ego is the Enemy, I realized that I had to read this one in the original, not others’ takes on it.  So I’m plowing through a translation.  Some days I only read a couple of paragraphs and ponder the emptiness of fame and the need to act according to my nature.  But this has had  a profound impact on my worldview and has prompted several moments of extreme gratitude  and understanding, even as it challenges me to be present and act according to my nature, no matter how others around me are, and to accept all things as happening while I observe and think and act, rather than feeling that things are out to get me, and only me.  At the same time, I ponder the differences in society between a Roman emperor and a random person two millennia later, and the impact of culture and societal structure on philosophy.  It seems easier to be stoic if you’re an emperor, and it rings a bit disingenuous to tell others to accept their lot as it is, when your own circumstances benefit from that.
  2. The Anatomy of Peace  And Leadership and Self-Deception:  The mind is a tricky thing to combat, and these books, focusing on the way of being and how we see and interact with others have also shaped my attitude.  I now check myself to see whether I’m “in the box” for people as I interact with them, and whether we’re locked into a loop where we each collude to bring out bad behavior in the other.  The mind is a very tricky thing….
  3. Happiness By Design:  As I define my life work again, and determine what this decade of my life will be themed, it helps to set up my life to support happiness.  Yet another book with ties to Kahneman’s Thinking Fast and Slow, which I adore, this book focuses on doing what makes a person happy and structuring a life so that the proper balance of Purpose and Pleasure for an individual makes happiness happen naturally.  In some ways, the equivalent of your mother’s, “You are the company you keep,” it’s also about working with how our brains are wired to achieve happiness and purpose.  My rate in this one has slowed down a bit – we’ll see how long it takes to finish it.
  4. Edgar Schein’s Humble Inquiry:   I swear I had a class with Schein while at MIT.  Anyway, this book reminds me to ask questions more and tell less, because telling is more condescending and true asking, where you don’t know the answer in advance, builds relationships.  Something I need more practice at, and I’m hoping this book gives me both the momentum and more tips for my toolbox.  We’ll see – just started it late last week and I’m already a third of the way through.
  5. Melville’s Moby Dick:  Somehow, I had never read this.  I tried to read it a few years ago and only got halfway through.  This time, I’m farther along and contemplating the nature of leadership and madness and obsession and the relationship between this big, wild planet and the insignificance that is man on a wide ocean.  Plus, the language, the writing is just so much more sophisticated than the junky fiction that forms a backbeat to what I read in a rush while waiting for kids’ activities to complete I’ve been reading both historical fiction and non-fiction about Britain in the Medieval period, none of it as well written as Melville’s tome. If you want to understand leadership, I’d go with the now-less-politically-correct Killer Angels, with its analysis of leadership on both sides of the battle of Gettysburg.  But Moby Dick has its moments too, even if those moments are interspersed with non-fiction details about whales….
    Hmmm, I seem to be on a leadership kick at the moment, and a focus on life priorities and choices.  There’s a theme there….

The Lost Art of Defensive Programming

When I was a programmer lass back in the days back when the internet was spelled ARPANET and www wasn’t yet a thing, I learned my craft from the teachings of giants.

I worked for a company that was very big on fault tolerance because stock exchanges and bank ATM networks trusted them with their money.  Things couldn’t go down for no reason, and you couldn’t lose data – that would cost money.

Even simple things like system maintenance and installing and configuring software had to work or be backed out in very careful ways, and often under very restricted conditions. There were windows for doing these operations, and they were carefully planned an audited, and any operation that resulted in an extended window cost money.

Other people’s money or stock transactions or phone calls or health information was on the line, so the companies that used these platforms were literally betting their business and their reputation on our reliability.

So I.learned to anticipate errors and protect data using transactions that were guaranteed, even in the face of failures, and came to understand that the transaction monitor was the heart of the business.  For maintenance operations and manageability, where I spent most of my time, I learned to anticipate problems, and make it very clear what was going wrong.  And I learned to be wary of conditions I’d never seen before and couldn’t imagine happening under normal circumstances  and make sure that they were visible and that the software failed fast and things were recoverable, either backwards or forwards.

I also learned the must basic, fundamental part of debugging – step through your code.  Either as a mental desk check (back when I started, it took a while to  trot over to the computer room to thread 8″ reel tapes so I could move new software to the test systems, even longer if I had to generate a new system image.  So I learned to read through and ask at each statement “Is this right?  What could go wrong here?”  Doesn’t take long for small changes, saves lots of time.

And then I learned to step through the code with a debugger (though, yes, I can debug with print statements too), watching statement by statement, to see that the values returned from calls matched my expectations, both good and bad. And to remind myself to ask, “What can go wrong here?”

Because that’s really the key for defensive programming. It’s not to ask, “How is this going to go in a happy-day case?” but “What can go wrong here?  How can the program detect it?  How can it report the anomaly so users can understand how to address it?  Can it keep going in the face of this situation? Or does this violate something fundamental so that execution must stop/reset?”

Why?  Because things change – memory and swap space fills up, network configurations differ, someone forgets to run as an admin user, system configurations evolve, networks hiccup, and on and on and on.  And things fail in mysterious ways.

The art of defensive programming seems to be lost today, especially in scripting and high-level languages that make it easier to just chain things together in a happy-day string of commands. “I know that if I split this string at the ‘>’ and take the next 8 characters then I get the current time in 24 hour format” only works if the output doesn’t change (I know, let’s prepend the date!) and someone doesn’t use the special character > for something unexpected and if there is an actual time there and it’s hh:mm:ss with no subsecond decimals and 24 hour format, and on and on and on….

I regularly encountering scripts that blithely assume that all is well when doing simple things, like copying files to nodes in a cluster or grepping to pull out the line that has the space available for a particular file system, or that if the command fails, the reason is a single well-known reason and the way to work around it is thus-and-so, no additional error checking required.  Just mush on. Any other failure gets blame-the-user, “You should configure things my way” (your /etc/hosts contains two different aliases for a server?  Should only contain one! And the first one MUST be the one this program is looking for!) or ignored.

Most recently, I had lots of fun when I got errors in a multi-step function right after a warning that said, “Warning – you might see errors in the next phase but that’s OK”.  I got errors, but apparently not the right errors, so I shouldn’t have ignored them because they were NOT ok.. As a user, how am I supposed to know what errors the author had in mind that are ignorable?  I call these fire alarm problems, as in,  “we’re testing the fire alarm between 10:00 and 12:00, so if you hear alarm bells, it’s not a fire.”  Well…. it might be….

Let me get all curmudgeonly for a second. “Move fast and break things” is cool and all, but it does NOT translate into “keep fixing one problem at a time until you teach your code not to barf on the errors that you’ve encountered in testing and first round deployment in the rarified air that is your small, not-scaled test environment.”  Code in the wild encounters plenty of situations you haven’t seen in testing….  And if you’re delivering a product or stack, dollars to doughnuts, it will encounter all kinds of things you’ve never seen before.

The easiest way that I’ve found to deal with this, especially in a dynamic administration or manageability kind of world is to code for the result I expect, rather than what I don’t want to see. Every place something can go wrong, define what “done” looks like and test for that.  If I’ve just created a file, does the file exist where I think it does?  If I’ve just written to it,  then not only should the write count be what I expect, but the file should have the expected data in it.

It’s really simple, and really hard at the same time.

In the end, there are really two kinds of checks – checks you add for debugging only (notice that I checked the success of my file write two different ways), and checks for unexpected failures that can happen any time.  The former checks can be removed in production code (checking the write count should be sufficient after the program is done, or even simply validated when stepping through via a debugger).  The latter should always be there (accessing the file you created a second ago might fail for any number of reasons, so check every single time). When in doubt, leave it in, and tune performance later.  Speed may be sexy, but only until you hit a wall:  Prematurely optimizing these out using an  “of course it works!” programming methodology inevitably costs more than the performance overhead of leaving them in.

And check for positives and what you expect to happen, not negatives since it’s easier to think of ways that things can go right than it is to be sure that you’re covering all the ways that things go wrong. Network names can be standardized differently,clustered logical servers might not have consecutive sever names or match the order of the alias names, clocks might not be synchronized, dates might be returned in different formats, filenames can have embedded special characters.  And those are the simple obscure bugs.

Finally, if a known non-blocking error can occur, check for it explicitly (down to parsing the error/exception info and ensuring that it precisely matches the signature you’re looking for – a file-doesn’t-exist error on the correct file name) and either suppress it or change it to an info message

And when something does go wrong, for the love of all that is programmatically kind and good, please include all the information you can when reporting the issue, ideally with a unique identifier for the location and invocation.  It will help the person trying to understand why a formerly well-behaved program has flipped out at 3am… and the person trying to understand the situation might well be you.


Teleworkers and Celebrations


I first started teleworking 19 years ago, and have worked out of my home ever since, in all sorts of configurations:

  • the only remote worker on the team
  • a team that’s entirely distributed
  • teams mainly located in 2-3 sites
  • teams on 2 or 3 continents
  • solo projects.

So I’ve seen a lot of successful and a lot of not-so-good configurations for teleworkers.

One of the challenges of being a teleworker is the issue of celebrations and recognitions.  It’s hard to find something individualized that fits each team member, and it’s easy to forget the remote contingent when planning the “We Shipped It!” release celebration, even when the remote contingent played a strong role in the accomplishment.

I’ve spent a fair amount of time looking for ways to do celebrations and acknowledgements for remote employees so I could reward my own teams.  There are some pretty decent ideas out there, but most didn’t seem to resonate with me or my software developer peers. So I started assembling my own list.

Acknowledging  teleworkers’ contribution to a big accomplishment with a line or two in a mass “Thanks, folks!” email message while the local contingent goes on a harbor cruise  or out to dinner or to a movie to celebrate sends a strong message.  Especially when that happens during a workday while the teleworkers are expected to just keep mushing along, like any other normal workday except with fewer chats from on-site coworkers. Yes, I’ve experienced this, including attempts to keep the celebrations quiet because remotes don’t feel left out of something they don’t know about.  Pro Tip:  They know and they do.

Seriously, if you’re fond of your teleworkers, that’s probably not the message you want to be sending.

And seasonal parties, summer picnics, and weekly wine-fests?  Sometimes seems like “Working remotely is reward enough!” so no need to even think about including remote workers in these celebrations unless they just happen to be on-site.

I’ve been there and done that on the receiving end.  Or been told, “Go take two hours and do something at the same time we’re watching our movie.” When presented like this, it’s not the same, not at all, and I sure don’t feel rewarded because it’s an isolating experience with no connection to colleagues, not a bonding experience.

For major milestones and even small celebrations, swag gets handed out.  Small laptop accessories, stickers, little mementos, flags, the occasional T-shirt, whatever.  Freebies for the employees and free marketing/branding for the company.  Remotes often get overlooked for these little items, because they’re not there for the party favors and besides, shipping things to them is a hassle.  On one memorable trip to visit colleagues on a different continent, we carted 15 baseball caps from that summer’s group picnic to a remote site where we had 15 employees on the extended team.  One of the people I was traveling with came from the main site and they brought them in their luggage.  Was there a hat for me or for the other five who worked remotely in the US? No, of course not.

Seriously, the swag problem is a no-brainer.  If you have 25 employees, then you get swag for 25.  Not 20 who go to the party and those 5 other folks who you never see but are voices on the phone and people who do pull requests and code reviews.  When the celebration is done, put the swag in the box and mail it, along with a “We missed you!  Too bad you didn’t get to see Avery bowl – 7-10 split Every.Single.Frame!” (or whatever) note.

Of course, the first and best way to include remote folks in celebrations is to actually include them.  Bring the team together regularly after a big milestone and hold celebrations then, along with planning for the next steps.  That way everyone’s there for the experience and the team is automatically strengthened.

But don’t hold off celebrating things, big and small, because “we’ll be getting together in a month.”  To be meaningful, the celebration has to be timely. When far-flung folks can’t be there for a timely celebration, be it a holiday party (any holiday’s a good time to celebrate) or a milestone, the show still goes on because that connects the team. So yay, we shipped!  Celebrate it!  And tell everyone that we did celebrate because building a culture that celebrates accomplishments and shares success is a good thing.  Put up pictures on a company board.  And ask the remotes to join in with theirs!

And that, right there, is the actual goal:  celebrations build team closeness.  Check your actions against that as the motivator, asking yourself:  Knowing what I know of this remote worker (you do take time to just chat with each worker, nearby or far away, right?), how can I use this to connect rather than exclude?

First, don’t hide the celebration and pretend it doesn’t exist.  Treating it as a shameful secret heightens the feeling of separation and second-class citizenship, whether or not there’s some sort of alternate compensation. If there’s a party or celebration coming up, let the remote workers know, “Hey, we’re all going canoeing next week to celebrate our release.  Friday, 2-5pm”.  At least they’ll know where everyone is.

And if it makes the celebration organizer feel a bit guilty to have this as the full sum of the conversation, well, that says that the remote worker should get something too….  So at the least, follow it on with “You’ve worked hard on this too, and we’re sorry you won’t be able to be there.  So take a couple of hours off to go see a movie or a ballgame or kayak on the river on us next week and send a selfie!” or something like that. If you have a display of pictures from these celebrations (electronic or on a wall somewhere) , make sure it includes the remote workers’ celebrations too.

This basic, simple format meets all the goals. Your remote worker

  • Had her accomplishment recognized
  • Had an opportunity to celebrate in a way that left the team still supported
  • Had an event that aligned with her activities, interests, and schedule
  • Had her celebration recognized just like everyone else’s
  • Gave others another topic for informal chat to bring the team closer together as people, not just voices on the phone

All with just a slight adjustment on the negative scenario above – in some ways syntactic sugar.  But also, with the pictures, actionable.

Sometimes generic ideas “Go do what you want!” don’t quite cut it, so here are some specific other ideas as well:

  • Dinner on us!:  A gift certificate for the remote worker and her family, which acknowledges the impact of the milestone on not just the worker but also the family.
  • Movie/Concert/Sporting Event on us: Again, with tickets for two and/or family.
  • Holiday concert:  In lieu of the generic holiday party or summer barbecue, maybe there’s a local seasonal music performance (Spring in the Park! 4th of July Blues Festival, Summer Art Festival).
  • Surprise!  It’s always been a simple morale booster to get a surprise package with thoughtful recognitions – a mug and some hot cocoa or coffee or tea for a winter celebration, seasonal food, simple things, and the seasonal party swag.
  • Coffeeshop get togethers: If your team is going out to lunch or a meal, end it at a coffeeshop or other IM location and invite your remote workers to join in then from a local equivalent, or even their own kitchen or back yard.  You can time this so that it adapts to different timezones – morning coffee, afternoon drinks, evening decaf.  Connect via IM or voice chat.
  • Togetherness via Technology:   If your team does monthly informal birthday celebrations or other similar milestones, put them on Skype or Hangouts or your favorite IM program, and everybody sing to the birthday folks!  If an in-office holiday party with caroling, send the music and join in the songfest.  Halloween? Everyone dons a mask and takes a picture then arrange a “who’s who?” guessing game.  Arrange for cupcakes or candy apples to be delivered separately, ideally timed for the morning of the party.
  • Personalized greetings:  Put together a chart or certificate with a personal acknowledgement and send it out.  Doesn’t have to be fancy or expensive, just “Thanks for making a huge contribution to the coolest project ever!” with screenshots, and then write a note, “Dave, the graphics wouldn’t’ve been this awesome without your attention to detail – THANKS!”  Frame and send. Or for local employees, do the same thing, but drop them off after hours, so when they come in the next morning, they have a “We belong to the team!” moment.
  • New toys: If the team is going on an active outing (picnic at the beach or park), send a related toy (flying ring, beach ball, whatever) or gift certificate (local cinema, spa, bakery), again with a “Create your own special moment!” theme.
  • Magician of the month:  When someone does something amazing or unique, send them a superhero cape or a magic wand or something to celebrate.  Ask them to send a picture back in their favorite place, with them doing magic things!  Highlight that in the group’s communications channels (slack, photo board, etc.)

Some people really don’t like to celebrate, whether on site or remote. Personally, I don’t like to expense personal meals, even when traveling, so telling me, “Go buy yourself lunch and expense it” just isn’t going to happen.   That’s OK, don’t force it, but do acknowledge their contributions and make sure they get swag.

These ideas really take just a little bit of thought, and even less action.  But it takes paying attention to detailsto make sure your far-flung team goes from good to gelled to great.  Because the team that accomplishes great things together also celebrates them together.  It just wouldn’t feel right to ignore Alex who lives in Minot, even if the rest of us are in Tallahassee – we’re a team.

The converse is also true: If it feels fine to celebrate without acknowledging remotes’ contributions, you’re sending the message that they don’t really matter.  And they know it.


Eyes on the Prize

In a confluence of events this weekend, I was reminded of the difference in focus between folks who are highly reactive and folks who have systematized a long term approach and align actions against that.  Each of these approaches has strengths and weaknesses, and I’m finding myself living in situations where I see the drawbacks and advantages of each.

First a bit of definition:  What the heck am I talking about?  Well, consider a situation where you’ve got a bunch of short term issues where you’re guiding other people, and where there’s an issue.  Maybe a new product and “what color is the print on the packaging?”  You’ve got a product team reporting to you that’s owns the product, and that team has a structure, and they’re working on on the package.  Meanwhile someone on the team other than the lead comes to you informally and asks that question “because we need to order that today.”  Without thinking, and because you want to move on, you give them a response.  And the person goes back to the team and says, “Decision made.  Pat says it’s blue #23.”

Most people see the immediate problem here.  You just bypassed and undercut the team structure you built and subverted the team.  Even if it’s just 2-3 people there’s a structure here and you’ve changed it.  And that means a bit of jitter on the team until the structure is redone. If that was your intent anyway, fine.  But if not, the next decision that the team makes is going to be harder and for no good reason.

Most people don’t fall into these traps for big decisions. It’s the smaller decisions where these sorts of things come up, nibbled to death by ducks.

And yet….

Sometimes these decisions should just be made and move on – the ROI on discussing every decision in the team context is painful.  Besides, some decisions really are time critical. And sometimes the lead or boss has to overrule for what appears to be capricious reasons.  And efficiency, especially in a startup, is a good thing.

But overruling too frequently, and “just because I was asked” creates a bigger problem, because doing it too often means that the team doesn’t make decisions easily (for fear of being overruled), and when they do, they have little sense of ownership “because Alex is probably going to overrule it anyway.”  The opposite of empowering.

The lead has to take the long view, but balanced against the number one rule of startups:  There is no long view if the short view doesn’t pan out.

Still… eyes on the prize.

If the goal is longer than right here, right now, and the team isn’t stuck and needing guidance, then it owns the decision.  The lead owns the structure, but the outcome?  That’s owned by everyone.  Because really, if you already knew what outcome you wanted, then why take the team’s time to rehash and potentially come up with a different result?  Like getting 5 people to agree on a pay-for-yourself dinner destination and then the organizer announces, “I have reservations and a coupon for a-totally-undiscussed-place so that’s where we’re going.  See you all in 15 minutes.”  Ummmmm, why did we just spend 30 minutes discussing something else?

So when someone comes and asks “What color do you want for the printing on the package?”, the answer is “How are you deciding?  Have you looked at ink costs?  Tested it? ” and other process guiding questions.

Last weekend, it was about an event and people wanting a decision from one of the people running it, without consulting the folks in the area that were responsible for that decision.  I was there and asked, What do the people in that area think?  And the leader said, “Oh, yeah!  We need them to decide this!  I’ll get back to you!”  Not because Bobby wanted to undercut the team’s ownership, but to get the decision made and get the interruption managed.

In another instance, it was about a process structure at our startup.  Startups are trickier than single-events because 80% done and alive is better than perfectly dead.  Still, when setting up a replicatable process,  you still need to be careful to put the bumpers up where you don’t want to go off the rails.  If customer facing means don’t execute commands on the customer’s system, be very firmly clear that when someone asks to “let me go in directly and run a few probes to see why the server failed.”  Not “just this once” and especially not before the (make it lightweight!) process has been tuned and owned by the team. Well, because there’s a reason that we ask customers to take action on their servers, not us – liability and safety and customer policy just requires it.

If a relatively routine decision stretches the structure near to breaking or similar situations recur frequently, there’s an underlying issue instead.  The real questions to ask are:

  • Was the structure wrong?   In this case, there is no “just this once”; it needs to be acceptable regular practice
  • Is there another issue on this team that forces team decisions to external folks?  Things like,  someone jockeying for position, deadlocked members, inability to find a time to discuss, unclear requirements.  If so, address that and give the decision back to the team.

A fully engaged empowered team owns the decision, and everyone on that team has a role in that process and understands the goals that were used to structure the process.  It takes a lot of hard work to build that level of trust and interaction and ownership.  It’s easy and fragile to undercut, especially early in the process or when stress is high.

So if you’re a lead or setting direction for a team that reports to you, be very careful when someone comes and asks you for a solo decision – are you undercutting what you’ve worked so hard to build?  Your first response should always be a question to yourself:  “Do I have to make this decision?  Right now?”

And if the answer is yes, be prepared to support the team as you pull rank.


This. At least sometimes, and when you least expect it

This article about the crap that women put up with every day and don’t even make visible because it’s so normal really got to me. I don’t go looking for being treated differently because of my gender. In fact others have to point it out to me at times because I’m pretty thick about such things.  So it’s gotta be gobsmackingly obvious before I even notice.  No females in any position of authority obvious, and it took me a long time to notice even that….

But if I mention something egregious, whether or not related to being female to a male colleague, friend, or family member, it’s immediately doubted and questioned:  “Are you sure?  Are you being rational or overreacting?  You know you can be a little too strong when you express your opinions, so it’s your issue to deal with.  Just rise above it and don’t say anything.”  and the like.

I’m reminded of Sheryl Sandberg citing the number of women who’d been warned about being too “aggressive” (aka bitchy) in performance reviews, as opposed to the number of men at GHC 2015.  Ok, men don’t get called “bitchy.” and the term “bastard” is almost a term of endearment.  As in, “the bastard got away with another one” said half admiringly.

I am reminded of the old posit that a lone woman on the team frequently finds herself with a minimal selection of roles:

  • one of the guys
  • bitch
  • group/room mom
  • temptress
  • invisible

I’ve been fortunate that on many teams, I’ve been a team member or team lead.  Not one of the guys, but folks don’t first think of me as a female to whom they have to apologize for swearing (my delicate lady-ears fall off at the sound of four-letter words) and don’t feel obliged hold the doors for me either .  Just how I like it(first one to the door opens and holds.  Easy peasy).

But sometimes when I least expect it, the gender thing hits me again, sometimes even with my normally well-adjusted colleagues.  I look around and I’m the only female in the room and there are 17 men. Or I interrupt someone who has similar group social capital and I’m a bitch, though I’ve not managed to complete a sentence without interruption.  Celebrations and thank-yous fall to me by default because “you’re so good at it, and I have no people skills!”  Told, “Oh, that’s just his way of giving you a compliment!” Or stares, though as a telecommuter, that’s been less of a problem.  I get rescued in weird ways, with a “lemme do that for you….” kind of thing.

And it’s a shock and I’m reminded again that it’s a crazy, unbalanced world we’re in.

The thing every single person, male or female, wants most is to be listened to and understood.  You don’t have to agree, just don’t say, “you’re wrong” or “Men have it much worse.”  Because, y’know, women leave this field at a rate 45% higher than men because men are masochists and enjoy having it much worse, right?

I will go far out of my way and give up sleep to listen to someone, hear them out.  If you’re on my team, that’s important to me.  Because, as I said before, I work with people, and people are very human.

Hat tip to a tweet from Rands (go follow his blog and tweets if you don’t already).

Process and Structure aren’t four-letter words

Why do people hear the word “process” and see themselves caught in a flood of molasses, moving slowly toward an ever-stickier goal? Why do people hear the word “structure” and think they’re going to be chained in place and performing jerky, robotic movements?

I try to school myself not to say these words. Not because I don’t believe in them. But because “them’s fightin’ words” for many people.

The moment I try to systematize something as simple as talking to a user, or plan for a second on how to engage users, if I use the “p” word or the “s” word, ears shut off and people hear 20-page status reports and if-then cases that run to 400 pages of instructions on how to brush their teeth or something.

No No No No NO!

I’ve spent many, many years as a musician, and one thing that I’ve learned is the value of jamming and the value of rhythm.  And that’s all Process and Structure are – the bones of the framework for the jam session.

Process is the rhythm around which the awesome project is built.  Rhythm can be heavy or light, but it needs to be there or things go nowhere.

Structure is the rules for what key to be in, the overall chord transitions, the tradeoff points between melody lines.   Structure can be heavily mathematically precise or loose and flexible, but, like gravity, it needs to be there or the jam session is just chaotic noise and wannabe soloists playing over each other as they jockey for position and attention or even just something interesting to break the boredom of noisy inertia.

My history is about not supplying too much structure or process.  It’s centered around light weight and trust in my teammates and shared goals that mean that none of us need overspecification and 400 pages of what to do if the cap is on the toothpaste or the lights are out when you get to the sink – you’ll figure out the best way.  Besides, the 400 pages of instructions don’t cover how to suppress new-vocabulary-word expletive education for the nearby minion when you’ve dropped the tot-lok magnet on your toe while trying to get the toothpaste out of the kid-proofed drawer. Oops, make that 401 pages of instructions. And counting.

Simple process: Remember to brush your teeth on the way to bed as part of the regular routine.  You know how.

One line and done.

When someone proposes a big, over-controlled structure to make a decision, the response shouldn’t be to pick it apart, and debate the merits of squeezing toothpaste from the middle versus bottom for hours on end, then move to rolling versus flattening as the tube empties.   Way too easy to get lost in the details and a permanent state of low grade hostility (“I am not even trusted to know how much toothpaste to use!”)  that substitutes for progress.

Instead, it’s back to what is the goal, and why is someone proposing that the toothpaste be kept in a drawer, neither in the front nor the back, but right in the middle, just left of center.  What’s the requirement here?  Off the counter?  Find in the dark so no brushing teeth with antifungal cream? These rules and over-detailed process specifications hide the bones of a very short, light statement that treats everyone as competent and allows for different brushing techniques.

How to extract that nugget from the overdescribed rules?  Here’s the simple, lightweight framework that I use:

First, are the goals clear?  Everyone know which end of the field we’re aiming for and what counts as a score?  In this case, WHY matters as much as WHAT, because you’ll never think of every.single what.  So the more every person knows what winning is, and how we know when we’re there, the more each one can test any decisions against “is this getting us closer to done and won?”

Second, what are the boundaries and what happens when we hit them?  If we all trust each other, where are the fences and issues where we need more visibility?  Is it when we do integration?  When we add new dependencies?  When the interfaces change? When performance testing shows significant difference from expectations?  What do those boundaries look like and what actions do we take when we get there?  Send an email?  Run some tests?  Ask for another set of eyes?  Put it under source control?

Third, what are some typical and hard situations where we need a structure or framework?  The typical, easy cases, well those are used to build the bones of the structure.   What do those workflows look like among all the individuals?  What’s the goals of each of their steps?  Those are the ribs of the structure.

The really hard cases are the ones that we test it with.  Within the framework do we come up with a way to handle the hard cases, all the way down to ugly details at ground level?  Does everyone agree that that way is acceptable, that even if there might be more than one possible way to cope, the framework drives to an acceptable means to address the issue?  We’re not looking for the best way here, just good enough and that the framework makes it easier to do the right thing and harder to do the wrong thing.  Honestly, when a real difficult situation comes up, it most likely won’t be exactly like the ones under discussion here, so the real goal is to make sure that the structure encourages the right process and acceptable solutions for things as they’re encountered

Fourth, what kind of communication and checkpoints do we need?  Even with connection points and fences and actions, it’s still worth checking in now and then to make sure that the framework doesn’t need some tuning.  These are the fine points of process.  I’ve found that demonstrations and discussions centered around actual use cases are much better than theoretical discussions.  Decide whether you need synchronization (meetings) or async discussions.  Use these lightly, and multi-purpose – tune process and build team – people are MORE important than the individual artifacts, because they’re going to be making many decisions outside the discussion, so these sessions are really about building understanding.  The goal is to have openness and trust that everyone’s working toward the same purpose, so these checkpoints are about making that happen more directly.  And “do we need this?” is always one of the first questions that we need to ask.  And adjust as needed.

This gives people the room to be themselves.  It means that no one is playing retroactive lawmaker, coming up with 50 steps to address yesterday’s minor issue that won’t ever happen again.  It means the team guides itself to efficient execution because there’s room to do it right and guidance on what “do it right” means.  And when that breaks down, there’s a way to deal with that which doesn’t involve acrimony.

Done right, process is as liberating and freeing and energizing as the best jam sessions.  People cover for each other and transition seamlessly and breathlessly, as a single living organism moving toward the same goal.

User Experience Driven Development – An Overview

I attended a talk last week about some pretty cool big data analysis. Part of it included a demo about how the conclusions were drawn – again, very cool  Unfortunately, the speaker also included the same comment a couple of times, “Of course, we need to  improve the user interface.”  Lots of words and a few boxes and highlights in what looked like a pretty standardized template, like the MadLibs games we would play as kids asking for a noun and a verb and a color and an animal and a food, filling them in and reading aloud a story about “One day I went camping in the chandelier  and we roped a red hamster and fed it pickles.  If you were familiar with what was being asked, you could get an understandable answer out.  If not, much hilarity ensued.

Now, first and foremost, I am a software developer.  I do a bunch of things, and I once wrote a post on another blog about how important it is for me that the project management pieces fit together as nicely as the actual software architecture, so I have a foot in both worlds.  But really, from the top I take a technical view of things, and while I am a writer, my favorite medium for writing is a software language, and my favorite entry mechanism is a programming editor.

So my favorite thing to do is to play with data, to build connections between different subsystems, to massage the data and the sources to make connections.  However the output looks. “We can fix that later.”  Or I could’ve said, “Of course we need to improve the user interface.”  Because I’m an infrastructure kind of person.

At least I was.  And then we tried a different way and got some pretty amazing results. And now I’m a convert.

The project was a manageability project, focused on applying data warehouse techniques to performance and manageability data.  Correlating log records, performance data, configuration data, health check output, and workload information to build a tool to allow at-a-glance management across the entire software stack and standardized across various data manageability frameworks.

We knew we had lots of data, but needed to turn that data into information.  Not much different than any other sensor analytics or BI problem, actually.  But with lots of objects to manage in a big data world, lots of workloads, lots of processes, lots of entities, there was no shortage of things we could do with it.  The real answer, was…. what makes sense?

The answer that we came up with was that we wanted to provide the tools for the various big data administrators to tell their stories.  To compare performance from last week, last month, last quarter, last year.  To predict when they would run out of headroom in their clusters based on workloads.  To see when workloads were outliers.  To be able to tell why things failed last night, looking at the log records and health check results from their Hadoop distro.  To handle the “Why is my query running long?” question when an user calls in after an ad hoc query – was it a disk failure or a crazy query plan?

That is, we had to focus on our users and what experience we wanted them to have.  How to make sure the tool fit their needs like a perfectly balanced hammer or screwdriver or saw.

So we started talking and walking through some scenarios based on our own past experiences in this domain.  Not individual use cases (though we sometimes did get to that level), but more workload tasks and “playing” with the data, working through “If I were debugging a problem, where would I look.”

This is trickier than it sounds, because each person uses different  keys to hint at what’s going on when they’re on a fishing expedition to figure out what’s going on.  Dietrich Dörner calls these “supersignals” in one of my all-time favorite books, The Logic of Failure. Frankly, I read this book, devoured it, really, and it changed my life in lots of ways. Not just for the projects I was working on at the time (not the one I’m talking about here) but also on the team dynamics and ecosystem in which we operated.  I still refer to my marked-up dead-tree copy, as well as the version on my tablet.  Amazing book.  HIGHLY RECOMMENDED….

Supersignals are each individual user’s way of dealing with complexity, “the existence of many different independent variables in a system.”  As Dörner says,

Complexity is not an objective factor but a subjective on. Supersignals reduce complexity, collapsing a number of features into one.  Consequently, complexity must be understood in terms of a specific individual and his or her supply of supersignals.  We learn supersignals from experience, and our supply can differ greatly from another individual’s.  Therefore, there can be no objective measure of complexity.

This is why one person looks at the log records first, then workload data for a particular job, and then the memory usage to discover an obscure problem with data ingest, while another person looks at transaction rates and disk IO rates before discovering the same thing.  And a third person looks at CPU rates and disk space usage and takes much longer to arrive at the same conclusion – it’s the set of supersignals they’re use to.

For situations, like this, you simply cannot prescribe a single precise workflow for arriving at at explanation of a particular scenario when you’re looking forwards at an issue.  (The answer is generally pretty clear with hindsight though).  Instead, the UX goal is to make it easy for each person to access data his or her own way, in the order she prefers to be able to suss out the factors that help explain and resolve the current situation.  Want to monitor logs by time?  Sure, easy.  Watching workloads throughput over time and for a particular application?  Fine! How about health checks at the server level ?  No problem – watch those too!

But sniffing out places where the issues are causing your antenna to twitch is only one piece of the puzzle.  The other piece is giving the tools to make it easy for you to tell your story (or, for this project, the big data administrator to tell his or her story ) to other people – other administrators, or managers, or even CxOs wanting to know whether there’s a big change in IT budget needed to handle upcoming trends, or when to predict the next quarterly rollup.

The UX for explaining these things to others is just as important as the UX for finding issues.  Both of these sides of the tool have to fit naturally with the way you think, like a chisel in a sculptor’s hand.

So we took a UX-driven design strategy.  We started with “what data is available, what tests could we do, what issues do people often see here?”  But we didn’t stay at that point very long.  Instead we jumped quickly to something very different:  We asked why would I care about this data? In what scenarios does this feed my need for information, and how  do I use it there?  Translated to different units?  Grouped across all activity? Isolated into a discrete chunk? Over time? One-time?  Associated with similar workload I recognize because they look like this? And so on.  What is that experience like?” is the question that drove us . Once we’d worked through that, the what of the actual implementation details was trivial.

This approach resulted in sketches and wireframes and mockups and “how do you connect the data?” and “when and how do you move from this topic to that?”  and “what kinds of charting make sense for this?” and other similar questions for anyone we could find who had any interaction with the subject at hand.  That meant lots of  fast user review and lots of informal feedback. Short cycles and lots of prototyping with show and tell.  And lots and lots of discussion about what might and might not work, where everyone’s feedback was critical.

As the wireframes and mockups were solidified, UXDD led us next into demos and discussions that invited people in, talking about how problems that they see would manifest themselves in different ways, sometimes ambiguous, so “that hints that we should go look at the event log to see if there is anything from this component.”   A radically different style of demo and discussion than what I’d typically used in the past, because suddenly it was real and personal and shared ownership, rather than instructions and canned scenarios and something akin to a scripted presentation spiel.

We learned a really important lesson here, one that was picked up from another great book, Sketching User Experiences:  Getting the Design Right and the Right Design by Bill Buxton.  The biggest advance was that we needed to have sketches when getting user input.  Not final, fancy drawings and screens with color and fixed positioning and fonts that looked like we’d invested lots of time in them, but obvious sketches and rough outlines.  If things looked too done, too formal, we found that we didn’t get real feedback and contribution.  At best we got “OK, that would work” or critiques of color choices or maybe small screen layout hints.  People didn’t want to criticize something that they thought was too “done” for their input to matter.

So we intentionally made things look sketchy for these discussions – no shading or detailed data, obviously fake-data graphs.  And especially not anything that suggested we’d invested a lot of time and effort in something  We’d bring up a drawing tool  and just start drawing circles and arrows and numbers and lines together on the fly, from a basic “nothing invested here, just to get us started” template. We didn’t want people to hold back and with a sketch, it seemed that they had no problems suggesting changes.  Sketches converted our discussions into a shared development/discovery process, rather than “can you comment on this?”. This actually turned out to be a pretty critical breakthrough for getting great feedback to refine our various screens.

In effect, we refined our UX for having a UX design discussion based on the style and depth of feedback received.

Those discussions also drove the infrastructure design for additional data acquisition, frequency, retention knobs, etc.  We took the thinking and feedback and worked backwards to figure out what kinds of data we needed, for how long for both decision making and retention, what it might be combined with and on what dimensions (not just temporally, but also things like workflow and cluster, and business grouping).  Then we worked to make sure that those associations were easy and natural to do.  Like breathing.

While the final details for the user interface for types of data or new services or new connections were being tested and finalized, we were busy hardening the plumbing and data acquisition.  Then, with it all integrated together, we could again examine the UX and play with the data to figure out whether there were additional ways to extract information from data, and talk more with our users to see what they could do now that they couldn’t do before.

This integration phase was the key, because it was only then that we’d discover disconnects between human thinking and what the cluster and its component entities did, everything from timescales to data that needed more translation before being human-friendly and query-read.  But the flexible architecture we chose made that level of change pretty easy.

In this way, the product advanced in leaps and bounds, with a breathtakingly fast turnaround.  And it led to fast understanding and a sense of ownership from our user community too. In the space of a half hour or so, it was easy to take users from “what is this?” to “can I have this now because I need it to answer the questions my boss just asked me about our resource usage, even though you didn’t show that exact scenario…”

Had we followed the more traditional use case driven design, “A user wants to debug a problem with a disk,” well, the result would most likely have been much less flexible. Do A, then B, then C, which will prompt action D and if that doesn’t work, step E.  And displays that, once you know what they mean, you can make sense of them, kind of like detailed weather models with lots of interpretation required, rather than intuitive understanding. These kinds of things are great with hindsight and with high predictability.  But many real-world problems aren’t like that, and it’s the hard ones that challenge you.

The technology in the talk I attended last week was pretty darn cool, but that coolness was lost in a sea of words and numbers, so that the important information, the actual value from the system, wasn’t visible.  It was technology but without an immediately compelling story that invited people in.  After the talk, I could see how I might use it conceptually but I couldn’t detail it out and integrate it into my big data worldview.  Why?  Because the interface made it appear that my involvement was entirely an afterthought, rather than the purpose of the exercise..

If the technology had started with the UX, and their related emphasis on why and how before worrying about the what of the technology, I could’ve been sold….  And for startups in this space, that’s the real key to traction.

Thanks to Gunnar Tapper for reviewing an early draft of this post.