Sunday, March 24, 2013

Complexity in software design


During last many years of my career in software development, I came across many systems. Knowing about all those systems and their designs was really an enriching experience but somewhere in the back of my mind, this question remained unanswered: What is an appropriate design?

Till now, I have come across with many designers/architects/developers/testers in industry and have heard from them about their approaches on designing a system against a given requirement set. Fortunately/unfortunately, I mostly came across with people who were extremists in their approach towards design. For some of them, design is nothing but a sequence of method calls and for rest it was always making things as complex as possible (in the name of futuristic, scalable, generic etc…)

I can safely say that these two extreme approaches can be labeled as under-design and over-design respectively. Both of these approaches may seem to be opposite of each other, but as far as repercussions are concerned, most of the time they lead you to very similar situations (surprisingly) like dump the software, redesign, rewrite, refactor etc. What actually happens depends upon up to what level of extreme approach was taken.

The challenge is how do you avoid doing the same or, identify if you are already in this trap? There is no generic answer to this question but of course, some of the checks and balances can always be put to ensure that you are not leading in the same direction.
  • Make sure that product is not too complex for users to use: Everything starts from here, if the product itself is not simple and beautiful, then there is high probability that you will end up having a complex design to build it. This is very costly and frustrating situation to be in as in this case, even if you have rightly skilled people to do the software development, when the product goes to the market, no one likes it, as it was too complex to use.
  • Understand the right use cases before designing: This is a prerequisite of software design. Never jump for design, if you are not fully aware of the right and complete use cases of the system. Knowing the end user workflow might not be sufficient and hence it is important to understand the use cases relevant for different components of the software.
  • Design against the real requirements not against your own intellectual dreams: Many of the times under the shadow of beautiful words like generic, scalable, modular, futuristic, flexible, designers end up defining a small problem into something which is overly complex to build. This not only complicates things for engineers but also creates delays for time to market, which could be potential business risk in this fast moving competitive world.
  • Design/implement in such a way so that others can understand: Even though it sounds strange but the reality is that it is more difficult to make simple things. For the longevity of any system, it is very important that others also understand it properly. If the system is too complex to understand then it is very likely that either it a case of under/over design.  
  • Technology and product domain both are equally important: Involve the right set of people while designing the system. It is not only important that the person is knowledgeable enough of the technology world but also aware of the domain, product requirements etc. It is a common practice that we assume that a person who has good experience in industry will do justice and with that assumption, we end up giving the design problem to a new senior person without ensuring that he/she is fully clear about the product requirements etc… In such cases, the person tend to prove his knowledge and skills in the whole exercise and tries to push most of the things which he/she knows about, which results into unnecessarily complex system.
  • Little knowledge is very dangerous: Sometimes, the problem is in hands of those people who do not posses enough experience and for them implementation of any system is nothing but a set of program statements partitioned into various methods/functions. This leads to a non-maintainable system where in any change in the system leads to a rewrite of the system. Hence it is very-2 important that the problem is given to the right set of people. There are plenty of engineers in the industry who possess little knowledge of things and pose as experts. If you hand over the problem to such people, their first attempt is always to prove themselves and in doing so, they propose to use tools, technologies, patterns which they have used. The net result is very complex.
Enough of dos and don’ts, let us go through some concrete example to understand some of the about mentioned points.

  • The requirement is to write a small program to reverse a string and without knowing the context of program usage, the person in charge starts assuming cases like
    • What if the input string is too big to fit in the main memory of computer?
    • What if the string contains double byte characters?
    • What if there is more than one string to be reversed. Etc …
  • You were supposed to design an employee management system and the person in charge of design starts thinking about
    • Will the database which contains the Employee table be able to handle more than 10 billion rows.. Come on, are you assuming the company to hire its employees from Mars?
    • What if relational DBs become a bottleneck for scale, so why don’t we think about the big data storage technologies?
    • What if the no. of concurrent request to the system goes beyond a limit and our normal HTTP server will not be able to handle it? Etc…
Without the context of the requirement and use cases, some of these what ifs may sound like genuine ones but in the given situations they are nothing but over complications. It is very easy to get trapped with these questions and companies end up paying lots of cost for hypothetical (so called futuristic) requirements. There is a very thin line between proper design and over/under design. Of course one should make things flexible, configurable, scalable etc., but up to what extent is the question, you need to answer in your own context. While designing the system, it is equally important to keep in mind the following aspects:
  • Long-term maintainability of the system.
  • Easy traceability: In case of errors, one should be able to trace back the faulty component easily. The components can be debugged easily.
  • Simple to understand for others to use or, change.
  • In case of malfunctioning, the system should be able to provide the right alert notification about it.
To end with one should always check against the following statement:
"When your design actually makes things more complex instead of simplifying things, you’re over engineering."

Time to say bye for now. In case you are interested in reading more on this topic here are some relevant references:

This book covers some of the items related to this topic nicely.