I originally wrote this last year on February 12, 2013. A lot has changed since then, including getting a new job. Yet I am still coming back to the same themes and concepts in my new environment. I feel like I’ve gained enough experience now to post the final draft on this topic, and the existential dilemmas & problems that we face as Software or DevOps Engineers. Even if you’re an Engineer of any kind, you’ll probably relate to these high-level concepts and philosophies, because they really have to do with solving complex problems of any kind in an impermanent world of change.

One day, a co-worker asked me this question:

Q: In the general case: Is it better to simply fix the immediate problem, even if I foresee a possible future problem?  And what do I do if solving the possible future problem creates more problems.  I feel like this question is practically philosophical…

It took me a while to think & write up an answer for this, but afterwards I realized that I had some helpful tips to share.

A: Ah, yes…  I am currently also trying to figure out the ideal solution to this philosophical problem 

Getting Lost in Problem-Space

Ideally when we encounter a problem, we would want to solve it without creating new ones.  However, in practice there are many cases where solving one problem either creates or reveals a new one (with some sort of cause/effect relationship).  Usually we can happily go along our way fixing each problem as it arises, uncover a new one, and start fixing that.  Hopefully this process leads us towards less problems and some feeling of completion of the task at hand where we can mark it as done.  However, sometimes solving one problem causes a cascade of new problems to arise. Sometimes also we can get lost in the maze of problems and we lose sight of the forest for the trees.

Increase Your Awareness of the Forest

In this case, I have realized that without a full awareness of enough cause/effect possibilities that stem from our decision, it is easy to get lost in the labyrinthine field of “problem-space”.  This can be very scary, discouraging & overwhelming, and give us many mixed feelings and beliefs about which way to go .  Sometimes this awareness of a potentially infinite minefield of problems can instill the ‘fear of the unknown’ which drives me towards the decision to do nothing, ignore, or procrastinate, but the initial problem still is not solved.  One helpful thing I’ve realized is that we all need rest sometimes, and a break can help to recharge & come back at the problem with full force, yet it’s important to do this consciously rather than self-sabotage or allow it to become procrastination.  We can realize the futility of doing nothing, so rather than “hiding” and withdrawing from engaging in life for too long, I usually end up deciding to do something with as much information as I know in the moment.  Sometimes I end up researching more to find some possible solutions or workarounds and choose the best one, sometimes I break the problem down into pieces and make a small step towards solving the first problem in the moment, and sometimes I ask for help or try to find some expert in the field who can either point me in the right direction or even fix the problem for me. Other times, I’m tempted with implementing a temporary or stop-gap solution. Still other times, I forget my own advice altogether and find myself stuck in a frustrated mood. In these times, it helps to have a friend and coworker to bring my awareness to this state, which helps signal me to take a break, or to pursue another avenue to help me to get unstuck.

Ideal Solutions vs. Quick Hacks & Band-aids

With solutions, generally there is some sort of feeling of confidence as to whether the solution is a quick hack, or a more complete one.  It’s good to trust one’s intuition in this case, but sometimes time prevents us from completing the ideal solution too.  Sometimes I end up searching and debugging down so many levels, running into so many problems and dead ends and poring over so much information that it becomes overwhelming. (Some have come to refer to this as yak shaving).  Over time, we would hope that we could somehow avoid dead ends and quick hacks, and there is some truth to the saying “a stitch in time saves nine”. I tend to prefer seeking the more ideal or elegant solutions in general, however, sometimes we all need some quick hack to get things working. In practice, sometimes we find that due to a deadline, some obstacle, or some other reason, we must implement some sort of quick workaround with hopes to come back and fix it later. The danger in this is that we haven’t solved the real problem, and eventually it can come back around and bite us again (and when enough users run into it, or enough people in the community are blocked by this issue, then a good general solution is important for all!). So again we have the spectrum of beliefs that could either lead us towards a quick hack or towards a more dependable and future-proof solution (although potentially more time consuming). In each case, I’ve come to realize that in the end, it all does come down to a quick decision in the moment of which path to take.  We can realize that the journey of getting there was all part of the solution, but staying in the moment in each moment feels important to me too. In realizing the power of Now, we can regain our balance and choose our path. The context of your present moment contains all the information you need to make the right decision.

Problem Solving Balance

It’s good to stop here and realize that there is a limit to the amount of knowledge and information searching that the human mind & ego can handle before getting tired and simply wanting to settle with any solution, whether or not it’s the best one.  Taking a step away from the problem, “rubber ducking” (ideally with a real person… “getting fresh eyes on the situation”), code reviews, or asking for help can provide what’s needed in the moment to become unstuck.  Hopefully these tools will lead us towards a cascade of solutions and the happy sense of completion I spoke of above .  It’s always a balance, and I’m always learning too.

Some feel that it’s best to avoid getting carried away in a “yak shaving” party, and then use this belief to justify avoiding all potential rabbit holes. However, the trap of avoiding all “yak shaving” hurts us because not all yaks are unnecessary problems to solve (seedefinition 1 here). If I encounter the same problem many times, it becomes a big enough issue that I feel the need to address it fully as its own problem.  In this case, it helps to do a bit more research and map out the problem’s “environment” and the “problem space” a bit in order to find a quicker solution to the problem.  Usually I can solve something and move on without encountering too many “dead-ends”.  However, sometimes I also find myself in a labyrinth of dead ends and getting frustrated with myself.  In this case, usually the problem is that I’ve made myself “snowblind” to the real cause of the problem in the first place.  Again, the methods of breaking it down, stepping back, getting help, or fresh eyes on the issue can help to get unstuck. Finally, In the software world there is also another very promising technique to prevent us from going down too many dead ends while coding: Unit Testing!

Map & Record Your Problem-Space

The easiest way to solve a maze is to put your arm out and follow one wall. You might end up going down a bunch of dead ends, but usually you won’t hit all of them, and you will always find your way out. The easiest way to avoid dead ends is to learn from your past experience and remember which way to go. But what happens when the maze is too big for us to memorize or we don’t remember the way out? Well, you’d probably want to create a map. In software, there is one good way to do this to avoid bug regressions: Automated Testing. The idea behind TDD is to write a test that will verify your code performs the function it should first, then write the actual functional code to get the test to pass. Over time, this idea allows us to create a library of Automated Unit Tests that verify our code works. It also protects us from old bugs recurring, or from introducing a certain set of new bugs as long as our tests are well designed. This method can help improve and streamline our coding cycle by immediately letting us know whether we’ve gone down an old dead end we already knew about, or if we have created a new problem. Essentially we are painting ourselves into the ‘happy path‘ which we will eventually converge upon.

The Philosophical & Existential Despair of Desire Alignment & Problem Solving

Because we are human, we do have desires. As engineers, we tend to be driven to solve problems & have a desire to do so. However, sometimes we don’t truly desire to solve all the problems we are faced with. Sometimes problems are too simple or boring to us, or sometimes we are faced with a multitude of small problems and issues which drag us away from our original problem. Sometimes we just feel so frustrated not knowing “Why things just won’t stay fixed?“. The answer to this question is that this is a world of forms which are constantly shifting and changing. You are not the same person you were 5 years ago, last year, yesterday, and even a moment ago. Software is constantly updating and changing, the applications and operating systems we work on are being developed and improved. Due to the complex dependencies and interconnectedness of these pieces of software, sometimes things end up in a (very) broken state. It’s really a multidimensional shifting puzzle that is constantly evolving over time. Think of some kind of pandimensional hyper-rubiks cube of entangled dependencies. Sometimes problems are too hard due to the number of simple yet interrelated problems. Here’s where workarounds and simplifying things can really help. Sometimes we may decide to give up, or find another way around the problem. Perhaps we may just decide to cut through the gordian knot, and avoid solving the difficult problem altogether. As engineers, and humans with an ego identity, we can tend to see these possibilities as unskillful, or perhaps undesirable. It may feel like giving up, however there is great wisdom in this route. It’s a perfectly valid choice to simplify a problem to the point of neutralizing it altogether. The real hard part here is our own internal struggle with our desires. Alan Watts as always has some wisdom on this topic:

Sometimes I find myself so deep in thought about something that my mind feels stuck in overload of the infinite subtleties of navigating life… yet surprisingly enough I find my way through even the most difficult of circumstances… and often enough I am able to see my thoughts in their “external” parallel thought form expression, as if looking in the mirror. This highlights the imperfections in myself which I fondly look at and love for what they are, in their own time and place. Then I remember that the axiom “as above, so below” holds as well… and it gives me great comfort in knowing that everything is alright, and I’m meant to be doing whatever it is that I’m doing, whether I see it as perfect or not.

“When you make a decision… (people have a great deal of anxiety about making decisions)… So when we decide, we’re always worrying “Did I think this over long enough, did I take enough data into consideration. And if you think it through, you’ll find that you never could take enough data into consideration. The data for a decision in any given situation is infinite. So what you do is you go through the motions of thinking out what you will do about this, and when the time comes to act you make a snap judgement. But we fortunately forget the variables that interfere with this coming outright, it’s amazing how often it works. But worriers are people who think of all the variables beyond their control, and what might happen. So then when you make a decision, and it works out alright, I think very little of it has much to do with your conscious intent and control.” – Alan Watts

Conclusion

At the end of the day, we all usually try our best to come up with good solutions, no matter how difficult or daunting this may be. There is a balance to be found between seeking ideal best-case solutions and implementing quick and usually temporary kludges (keep in mind there are elegant hacks too!). Also, it’s important to note that perfection is an illusion, because perfection is highly subjective (things can always be improved, or be worse). It’s ok to settle for “good enough”, as long as you take an attitude towards continuous improvement. An positive attitude of accepting that mistakes can and will be made (and that’s ok!) with the habit of learning from mistakes creates a direction of evolution towards continuous improvement, while a self-defeating attitude of overwhelm with things being imperfect can lead to an attitude of giving up. Sometimes it is very overwhelming to get lost in a labyrinth of problems. Therefore it can be quite helpful to learn some mental philosophical kung-fu and other techniques which we can use to regain our balance. This is possible without going too far into reviewing the entire spectrum of Agile Software Development philosophy, or too far into software-specific patterns or techniques. The most helpful techniques for Software newcomers are those that can help us feel less overwhelmed and help to re-frame the situation such as: Research, Breaking the Problem Down, Getting Fresh Eyes, Asking for Help, Seeking an Expert, Mapping your Problem Space, Workarounds, Simplification, and Cutting the Gordian Knot. The most helpful technique is to realize the power of choice in every now moment, and mastery is knowing what technique each moment calls for. Over time, we learn lots of techniques and eventually become a master or expert in our field. Every teacher was once a student, and the best teachers are those that learn from their students. I’m always learning more and working on improving too 😉 Happy coding!