Software Development - Creative Process*
Software Development as Learning*
Importance of Experience*
Father Structuring of the Software Development Process*
Putting the Structure Together*
Not Obvious Organization*
Improvements of Productivity*
Structuring the Software*
Unification - Libraries and Toolkits*
Simultaneous Development of the Set of Products*
Software is one of the most complex and innovative products.
Specifics of the Software Development are caused by the high level of creativity and high speed of the introduction of new ideas in the software industry.
This in turn is caused by the low entry price, low production cost, and short initial production phases in this industry. As a result, the software industry operates in the environment of high pressure to innovate and introduce new products.
Software Development of big corporations or funded by the government is relatively shielded from this market pressure to innovate, and it pays the price - results of such development are often unsatisfactory. Bigger and bigger parts of projects a while ago done "in-hose" now are done with software products produced by third parties.
This is a sign of maturity of the industry. The availability and quality of the products allows purchasing the products-parts on the market instead of their internal development (instead of vertical integration - reliance on the market).
This new phase in Software Development brought new problems.
In the software industry, managers are after specialists who have experience in the particular area, where managers are involved currently. Or they hire fresh after school specialists, who will learn and follow manager's way of development.
This approach looks reasonable, but it is self-defeating from the industry point of view. It creates artificial shortage of the software developers and, therefore, a rising spiral of wages. The area of Software Development is so diverse and moving so fast that for any more or less innovative project there are only a few specialists available, which have experience relevant to this project.
This approach of managers creates a pathological reaction of Software Developers - they are looking for places where they can gain some (superficial) experience in the new "hot" area in a short period and move to the other place for better salary.
It is actually dangerous for them to stay a long time in one place - the software industry moves so fast that newly acquired experience is becoming outdated with each day.
In the current work setting developers rarely can learn new subjects, which are loosely related to the work. The only way to stay up to date is the perpetual learning after the work. It is difficult with the usual software developer's 50 hours workweek.
In addition, employers often cannot verify that the potential employee has needed skills - they do not know exactly what kind of skills they need, because they need new skills. Hence, developers who can better present their skills have better salaries, than ones that work better, but cannot present themselves well.
Recruiters are definitely winners in this situation - huge turnover of workers, tough and fuzzy requirements of employers, and hardly verifiable skills of workers are good environment for brokers on the labor market.
This combination of factors produces high overhead - subtract the time the new developer is learning in the new place and the time when the developer is looking for the new job, and not so much is left for the effective work.
Software Development - Creative Process
There is one fact, which confuses many. Actually, it takes a month of learning and a year of training to become a decent programmer. This is similar to any craft. However there is more in Software Development than simple programming.
There is an old dream of Software Development managers to make Software Development similar to the work on conveyer. This dream still supports the demand for a slew of case tools and code generating tools, which suppose to automate the development and make developers exchangeable. However, all tools, which stiffen creativity, usually backfire - the workable results cannot be delivered in time. The regular failure of these products should drive home already the thought that Software Development is a creative process, and should be managed as a creative process.
It is time to recognize software industry for what it is - an industry of high risk, high degree of innovation, high reliance on innovative abilities of individual developers and high pay-off in the case of success. It is similar to ... the entertainment industry.
Managing in this environment is not an easy task. Two contradicting goals have to be pursued:
We can learn how this can be done from the successful software companies. They have some things in common:
Currently, many software companies use quasi management process.
Usually, managers know even less than developers the difficulties of particular stages of development. They collect this information from developers (and developers are tempted to mislead managers in the process). Then managers create tough schedules of development, which cannot be met, and everyone knows it. These schedules are perpetually corrected as managers learn more about the specifics of the project.
In other words, managers learn the process of the development of the particular software product during the process itself and they use the moving schedule as the learning tool. The idea is cute, and it works. The only problem - it produces moving deadlines (sometimes vaporware), tension in the software development team, and utterly frustrated customers, who cannot plan properly the deployment of software products in their business.
No one actually tried to estimate how much better would be customers of software industry, and as a consequence the software industry itself, if only the uncertainty of the introduction of new announced products and their versions could be reduced. We think that the economy as a whole has mach to gain from the better order and predictability in this area.
After all, there is already in place one powerful mechanism of dealing with uncertainty in this fast moving industry - the chain of versions of the same product, for which customers gladly pay. It is irresponsible in part of software companies to pass the uncertainty of the development of particular version of a product to customers.
It is in everyone's interest, if the delivery of the next version of software product is more certain, than it is now. However, it is not a part of the industry culture yet.
Software Development as Learning
Many specific features of Software Development process can be presented uniformly, if we look at it as a learning process. Above we presented the moving schedules from this point of view (managers' learning tool).
The difficulties of Software Development process arise from the high degree of uncertainty. Everything is changing and changing fast:
An anticipation of change and maintaining a high degree of preparedness to the kind of change, which one cannot anticipate, is a substantial part of Software Development process. Hence, learning has to be an integral part of this process.
The tested in many industries process, which blends learning and product development, is implemented as a chain of corrections-improvements.
It has an initial phase, which sets the stage for this chain and prepares some simplest variant of the product.
It has corrections-improvements steps, where
After initial stage we have a variant of the product, hence, we can initiate the steps of its improvement.
This process allows us to stay close to real situation, and it allows us delivery of the product in time.
Everything we know about the product and the process of its development we should used up in the initial stage.
Each step of correction-improvement should be based on learning.
Initial steps of different projects vary.
For example, if we had done similar projects before, we can start with the development of software libraries, interfaces and test environment.
However, if the project is new to us, we implement first a simple part of the project, and we proceed with development and arrive to something actually working as soon as possible.
The idea of the chain of corrections-improvements is widely used in the product development in many industries. However, software usually is developed in much more uncertain environment and in the process of corrections- improvements, we should address some unusual issues:
Hence, usual in other industries division of development process on stages does not work well with Software Development process - everything undergoes corrections simultaneously: specifications, general design, software design, testing system, design of tech support system, etc.
If managers and software developers learn simultaneously during the same process, we do not have proper specialization, proper division of work (from which management as a discipline had emerged). This division of work was helpful in other industries; can we achieve similar results in Software Development?
In the case, when there is a developed already prototype, or there is an accomplished similar development done in other place, managers can reliably plan and organize activities. Otherwise they do not have enough information to do their special work.
Hence, if we want to use tools of management in Software Development, we have to divide Software Development process into two processes running in parallel. One is the process, where developers learn, what can be done and how much it takes. And the other one, which is based on the results of previous one, where reasonable planning and scheduling can be done.
This brings the need to code again in the second process, what was coded already in the first one. In the development process in other industries, not software industry, this would be viewed as a major flaw of the organization of the process. However, in Software Development this is actually an advantage. The second coding (production coding), is done when there is already some working variant of the code in place; when all ideas are settled already, all problems of development are solved. In these circumstances, coding is easy, it can be done quickly, it can be tailored for easy testing, and the result of coding has higher quality than usual.
The trick is not to succumb to temptation. We should not delegate this second, production coding to less skilled programmers. The same programmers, which participate in the first, development process, should dedicate some, relatively small, amount of time to code in the second, production process. However, it is possible that a programmer different from one, who programmed a particular part of the code originally, does production coding of this part. This arrangement assures high quality of the production code and gives perpetual feedback to the first, development process.
This separation of Software Development process into the initial development process (Core process) and the Production process has some additional benefits.
During the Production process we can achieve different goals, than during the Core process. We should produce the code, which is easy to test, easy to maintain, easy to explain to others, easy to "walk through".
If programmers, who code a particular part of the software during Production process, are different from ones, which code it during Core process, than this produces an interaction between programmers and clarification of the code.
Because coding for the Production process is so easy, coding of the same piece of program can be broken into short stages with clear requirements. Each stage can be done by a different programmer (or a group of programmers). We do not have this luxury in the Core process, where there is a need for careful study of particular area of development.
We still have to address one thorny issue of perpetual learning of software developers. Software industry is moving fast, and software developers have to follow, if they want to stay employed.
This situation does not exist in the majority of other industries. Usually knowledge acquired in the college supplemented with experience at the work place serves a long time.
It is different in software industry. The situation is exasperated by the fact that to learn new ideas a software developer has to program and try them, often in some special software environment. New ideas can be in the areas completely unrelated to the area of current developer's work.
Nevertheless the knowledge of these new ideas can be extremely useful - it can save months of development time, or open up new possibilities.
Software companies, with all their talk of the support of learning of software developers, do not look kindly on developers venturing into the areas unrelated or loosely related to their current work. Developers still venture, but quietly, and this produces unhealthy environment.
The other solution, which developers find in this situation, is hopping from one employer to the other, learning new things on the way. It is stressful, and we had shown above, that it is destructive for the industry as a whole.
We see the only reasonable solution in this situation - recognition and encouragement of developers' venturing in the areas loosely related to the Core process with some small personal projects. The results of these projects should be discussed - this helps in the learning of a team.
This how the second separation comes - in addition to the Production process and the Core process we define the Exploration processes, which run concurrently with these two processes, and consist of small personal projects.
Exploration processes should feed the Core process with new ideas and new hands-on experience.
Importance of Experience
Managers in the software industry have a hard time as is, and experienced software developers, who are quick to point out managers' inevitable mistakes, are more than annoyance. These types are tolerated, but the manager would rather hire a recent college graduate, who can learn "the company way" than an experienced developer, who brings new ideas and a lot of "trouble".
However, experience in Software Development is extremely important. Only experienced software developer can anticipate future changes of the code and produce easy to maintain code. Only experienced developer delivers the code with many reusable parts. Only experienced developer designs software, which is easy to debug and test. All this has a direct effect on the "bottom line" - the time of development and the quality of product.
Our division of Software Development process into three concurrent processes - Production, Core and Exploration, increases the level of experience of the developers, and simplifies an interaction between developers and managers. This affects results positively.
Father Structuring of the Software Development Process
In many industries over the years of experience was developed a sequential structuring of the development process, where different specialists are involved on different stages of the process, for example
This division has its weak points, but benefits of specialization usually outweigh them.
In Software Development, we do not have this luxury - the environment is changing too fast. We have to include in the chain of corrections-improvements all elements of the development process; they have to run in parallel.
The elements of Software Development process, which have to be included in the chain of corrections-improvements, also include processes that are usually classified as marketing.
However, some elements of the development process stabilize in the process of correction-improvements sooner than others, and this makes the process manageable. This justifies the consistent division of the development process on concurrently run interrelated sub-processes:
This structuring in the bigger or lesser degree is applicable to any part of our complex development process - to Exploration projects, to the Core process, and especially to the Production process.
Putting the Structure Together
We have to combine quite a few nontrivial ideas and arrive to something workable.
The basic idea is the development process as a chain of corrections-improvements. This idea allows us to learn during the development, and it keeps us close to the reality. We apply it consistently to all development processes. We encourage its application for the Exploration projects, and we organize our Core process and Production process according to it.
Our structuring of the development process is important to the Core process and Production process; however, we make emphasis on different elements in these two processes. The Core process is dedicated to the development of the concepts, design, and variants of the code. Production process is dedicated to the delivery of the quality product in time; hence, more emphasis is on code analysis and testing.
Now we have to bind all this variety of processes that they all work in concert, and produce what we want them produce.
We have to take a broad look at Software Development process. It is a process of delivery of the product according to our ability to produce it and market ability to absorb it.
We start with the Core process. After a few cycles of corrections-improvements, we have a pretty good picture of what we can produce, and how it can be presented. We explain it to marketing, which starts the learning process of how it can be sold. We, in turn, start the Production process.
The Production process is controllable - we can deliver variations of the product, and we know when we can deliver them. We should never delay the delivery, because we have so much reserve - in the case of emergency, software developers set aside all their Exploration projects and the Core process and concentrate on the Production process.
We can use the Production process as a tool of market penetration. Marketing affects defining of the specifications in the chain of corrections-improvements of the Production process, that it is possible to try a few variants of the product and find, which works better on the market.
Our structuring of the Software Development process created a selling tool!
We offloaded, at least partially, the pressure of the deadlines from the Core process and now we can benefit from it. The goal of this process is
The better is the code here - the less time developers need to dedicate to (actually boring) Production process.
The Core Process is organized in the way a Software Development process is usually organized, but in our case the pitfalls of such organization are irrelevant - we will rewrite this code. We can afford the encouraging of the different individual styles of programming, we can switch the work on the particular piece of code between different developers, that they can learn each other ideas, and improve the code on the way.
We expect developers to test their ideas on small personal Exploration projects before they implement them in the Core Process. This organization leans toward highly modular code, which is our goal, and which is difficult to achieve with other organization of the development process. Also, it defines the major part of Exploration projects, which are automatically connected to the Core Process.
The perspective of writing many different variants of the code based on the code developed in the Core Process orients developers on the design of the reusable code, which is an additional benefit of our organization.
Production process serves as a "reality check" for the Core Process. Production process has strict schedules, and if there is not enough quality code developed in the Core Process, the code has to be developed independently in the Production process under the pressure of deadlines and heavy testing.
Production process is the process bearing visible fruits of our organization of Software Development. This is a process where tools of management can be applied successfully:
This process is a set of strictly defined Small Production Projects developed independently and coordinated with the strict schedule. Small Production Projects do not take long time - simply some of them deliver (strictly defined) intermediate results.
The same software module can go through the chain of Small Production Projects and developers. Development, integration, testing, correction-improvement - everything is driven by a strict schedule. As a result, we have a final product in time.
Specifications of the Small Production Project are never changed. If there is a need in a different result, we launch a new Small Production Project. The original Small Production Project we finish any way and we "shelve" results.
Corrections-improvements of the results of the Small Production Project we perform as a new project.
For the duration of the Small Production Project, developers make temporary groups dedicated to this project. One developer can belong to a few such groups.
We call it "atomic" process.
This type of development automatically produces simple and optimally documented programming style. The same code goes through many hands, this encourages some degree of the uniformity in the coding style, and comments explaining unclear parts of the code accumulate gradually.
In this environment, the code analysis by other developers is easy and natural; it should be a part of the process - it greatly improves the quality of code.
Not Obvious Organization
We had achieved important results with not obvious organization of Software Development process.
We insist that
All this for the person who did not struggle through Software Development process can look as "waist"; this is why it is so non-trivial.
Actually, we insist that with our organization we deliver better product with smaller resources and in the shorter time.
We achieved more with our organization:
Improvements of Productivity
There are plenty of publications dedicated to the Improvements of Productivity of software developers. However, the majority of the methods advocated there does not work in reality. Here we summarize a few ideas, which actually work.
Structuring the Software
The optimal structure of the software often is different from the structure of the design of product. It reflects factors as:
Awhile ago the major goal of the software structuring was the effectiveness of the program - the speed of execution of the program, the size of the program and the size of data it uses and generates. With rising hardware capabilities, software complexity and degree of innovation other factors became more important.
Now it is more important to have software, which is structured the way we think, not the way the computer works. Still, there are problems of the effectiveness of software, but now they are solved with the design, which localizes special parts of the software crucial to effectiveness, and only those parts are developed in the special way - optimized for speed, size, etc.
There are many ways to structure software:
However, it is practically impossible to teach how to do it - this work requires real experience, the deep understanding of the software application (and hence, its future corrections), available tools, and patterns of the software design. This is a highly creative and subtle work; it cannot be done by the committee, and only rarely can be done by the small group of specialists having similar ideas. Mostly, there is a need to have one designer, who gathers all possible advice, but creates the consistent design, based on ideas, which he often hardly can verbalize. This designer has to monitor the development process until the end of the development - often well-minded local improvements go against the overall idea, and only the designer has the clear picture of this idea in his mind.
Software Development requires a lot of concentration and creation of special images in the mind of programmers, which keep things together. However different people think differently and it is difficult to share these images. Hence, tight interactions between programmers during the development are difficult. In addition, different programmers follow different programming style and there is no sense in stiffing their creativity by imposing on them style restrictions.
There is a need in the separation of the development area of different specialists.
This can be done with the proper structuring of the software and creation of relatively stable "interfaces", through which these different parts of the software interact.
The main feature of the system of interfaces is its conservative nature and consistency. This is a place, where programming style restrictions are appropriate.
During the development process interfaces change, hence, it is important to have someone in charge of all interfaces, that their consistency is preserved.
Interfaces allow fast development of semi-functional modules, which other developers can use for testing and debugging. This allows a parallel development of software.
Unification - Libraries and Toolkits
Specialization is a great invention. It allows higher quality and faster production. Specialization found its way in Software Development process a long time ago. However, specialization has its weak point - groups of different specialists invent independently different solutions of similar problems.
This phenomenon is so well known in many industries, that there are specialists in unification, which fix this problem.
They come on the final stages of the design, look at the project as a whole, find modules, which perform similar functions and try to replace them with one separately developed part, which is used everywhere. In many cases, this simple technique leads to great savings and improvement in quality.
Experienced designers keep this in mind, and design the set of universal parts simultaneously with the product design. This brings much better results, than ordinary unification, and often it leads to shorter terms of development.
Similar situation exists in Software Development.
The set of universal parts in software includes libraries, toolkits and application frameworks.
Some of them are so universal, that they can be products on their own. However, many of them are finely tuned to the particular business or even particular project, and they are used narrowly. Their design is difficult, requires substantial experience, skills and dedication.
Development of these libraries goes through the same chain of corrections-improvements. This process is driven by the requests of developers, who use them. On the way, it is easy to lose the integrity of the style of the library. Hence it is important, that there is someone, who is in charge of the library development, and who can assure the consistency of the style of library interfaces.
Here, as in the case of other interfaces, which have to be used by many different developers, the consistency of the style is important (while the consistency of the style of the library internals is not).
The dedication of the resources to the library development is often overlooked, however the benefits of the library are many. We enumerate a few of them here.
In the beginning of the library development, there is some knowledge of what can be a part of the library tuned to the given project. This can be developed and offered to the developers. As the development process proceeds, the library developers learn new needs, and expand and correct the library. This way the library emerges from the development process. It is automatically accepted by the developers, because it addresses their needs and it does not impose restrictions on their creativity. As a result, this library is non-trivial, it accumulates experience of the development.
This library (toolkit, etc.) can be used for future developments as a valuable source of accumulated experience.
There is a widely held myth that libraries are tools of the code reuse.
Y2K bug should show to everyone that there is no such thing as a straightforward code reuse. Y2K bug is a result of the programming style, which was effective in its time. Simply the code should not be "reused", but rewritten. What was effective years ago is not effective (even not useful) now.
The programming, which is effective today will not be effective or may be even will not be acceptable a few years from now.
A perpetually updated library is a tool of accumulating programming concepts in a coherent system. This is a source of its power.
Simultaneous Development of the Set of Products
In many industries, the design of different products goes independently.
Much better results are achieved, when the entire set of products is developed simultaneously, when the possibilities of unification are taken in consideration on initial stages of the design. This is especially useful in Software Development.
Bigger resources can be dedicated to the development of libraries, toolkits, etc., when the set of related projects can be looked upon as a whole. This development is a difficult undertaking - it requires a good understanding of the needs of particular Software Development.
There is always a temptation to approach it from the theoretical point of view with libraries, which offer extended functionality and have "right" interfaces. This practically never works - developers do not use this kind of libraries. The useful libraries have limited interfaces, which easy to understand from the first glance and memorize, have some special functionality, and do not address the issues irrelevant to the particular Software Development. These libraries have to grow out of the particular development and be based on the particular experience.
There is a trend in the Software Development from the efficient programming (small and fast code, small storage space) to conceptual programming (design and code is easy to understand). The efficient programming has its place, but it is localized to special software modules, which determine speed and storage space the most.
The cause for this trend is expansion of the possibilities of the hardware and high speed of innovation in the software industry. Developers have to communicate well between themselves during the development, and the code have to be easy to understand, when there is a need to correct it during the maintenance phase.
We think with concepts, hence if we want the design and the code to be understandable, we have to present it as a construction of concepts where the complex concepts are build with the help of simpler ones.
The need for the conceptual programming is learnt through many failed projects - people need to communicate well to take the project through many steps of Software Development process.
The development of libraries (toolkits, etc.) goes in the same direction of conceptual programming. To develop a useful library one has to present the project as a system of concepts.
Conceptual programming makes understanding of the code written by the other developer, or written by oneself a while ago, easier.
Conceptual programming requires intense communication between developers and study of the current state of Software Development in general - concepts emerge as the product of society, including given group of developers. Sometimes it is better to use the design, based on the well understood by the group concepts, than the more efficient design, which is based on concepts new to the group. Concepts are tools of the consensus.
Conceptual programming allows delivery of the reliable code through the development process, which is not based on the exhaustive testing (which sometimes is impossible), but with the clear conceptual model, which exists in developers' minds.
The technique of defensive programming falls in this category. Defensive programming requires each software module or layer to check the input for any possible inconsistencies. This allows creation of the simple model of the possible software faults, which allows simple design and testing of the software.
Conceptual programming has its requirements. Specialists have to work with the project for awhile before they arrive to the useful for this project system of concepts. This new system might contradict to the original system of concepts, with which the design had started. It means, that the project has to be rewritten, particularly some perfectly working code has to be rewritten. The dedication to the conceptual programming means that there is a mind set and a time allocated, which allows such rewriting.