Some people still don't believe that open source projects can be better managed than proprietary [1] software projects. But the evidence, and developers' daily experiences, prove it. This document tries to explain why.
Proprietary projects can learn from this. By managing their projects acccording to similar principles they can improve quality. If they find it impossible to simulate these conditions in a closed environment then they should take the short-cut by participating directly in the open source community. The overall message here is that open source projects are successful because they are subject to such intense competition but much proprietary software fails because it is not.
These points apply to all proprietary software development, but they are particularly significant for in-house software development that doesn't result in a mass-market or resellable product. That is the vast majority of proprietary software development today.
These points don't apply to all open source software projects, but they apply to the successful ones.
Many of the examples here refer to the GNOME project, just because it's the one that I know the most about and because it has formalized its release process more than most projects.
[1] Notice that I use the term "proprietary" rather than "commercial". There is no reason that open source software can not be commercially viable. In fact, there are many proprietary software projects that would be more commercially viable if they were, at least partially, open source. Sooner or later you must decide to aim for less than total market monopoly in order to allow yourselves at least some market success.
Motivation
People work on open source projects because they enjoy it. These happy developers are productive developers. Managers of open source projects must ensure that the developers feel valued and fulfilled. They must minimise the tedious aspects of the work to ensure that development remains interesting. Otherwise, projects fail.
Although money can provide some incentive it does not provide as much. Managers who say that money is the greatest motivator are justifying their own poor performance. Managers of proprietary software, just like managers of open source software, must ensure that their developers are motivated properly. It is not enough to think that they
should feel motivated.Standards/Consensus
Open Source projects must conform to, and reuse, accepted, up-to-date standards. Proprietary projects, without the benefit of high visibility or feedback are free to make inferior decisions.
Open Source developers are usually a more representative cross-section of the development community. Therefore they can better judge and express the true technical consensus. In contrast, corporate recruitment uses arbitrary targetting and generally results in a sample size that is statistically insignificant. Proprietary software projects must encourage much wider involvement to create similar conditions.
Application Programming Interfaces separate different parts of software, such as libraries that are used by applications. They allow software to be divided and simplified and reused.Clearly separate modules
Successful open source projects usually consist of modular pieces with clear interfaces (APIs) and dependency chains. Work is divided among specialised, interested, groups. Successful projects, must be targetted and well-defined to attract developers and users. Therefore they often restrict themselves to solving specific problems.
Due to their large audience and high visibily, open source projects often spin-off sub-projects so that parts of their functionality can be reused by others. Because each API is used by many projects, in many different ways, the API is thoroughly tested and criticised, inreasing the chances that it will be appropriate for yet another project. Developers know that APIs should be loosely-bound but open source APIs are loosely-bound by necessity.
Simultaneous API stability + API development
Many proprietary projects suffer from one of the following problems:
Unstable APIs.
The API (class declarations and method signatures, for instance) change frequently, causing runtime and compilation problems for software that uses those APIs.
Stagnated APIs
The API never changes, in order to avoid disruption of dependent software, so development ceases apart from minor bug fixes in the implementation. Major architectural problems, though well-known, are never fixed and eventually the outdated design becomes inappropriate in the modern world.
Developers often fear that they can not solve one of these problems without causing the other problem. The solution is to maintain both a stable and an unstable branch, both of which can be installed, used, and developed, independently.
Ideally the new unstable branch should be created after the previous unstable branch has become stable. These 2 versions of the API should be able to co-exist on systems so application developers are not forced to use the new API until they are ready. For instance, C or C++ libraries should have different library names and separate sets of headers for each major version/branch of the API.
This lets old projects continue to use the stable API, while new projects, or new versions of projects may choose to use the new unstable API. They can benefit from improvements while tolerating some changes. Though not all projects will use the unstable API, those that do will provide valuable feedback. So when the unstable API finally becomes the next stable API, the developers can have some confidence that it really is ready.
If you don't know what an API is (as opposed to an implementation), please find out and then read this section again.
Some recent examples: GTK+ 1.2 and 2.x, libxml 1 and 2, GNOME 1.4 and 2.x
True market pressure means pressure from outside. Internal projects have no competition because no other company is offering a solution to the same problem. To encourage competition companies should either
- Encourage separate parallel development teams within the same company.
- Look for generic software used by other companies with similar problems. There are likely to be multiple candidates, leading to competition.
- Open-source the reusable parts of their own software and encourage other companies to do the same. This is obviously easier for software that is not very specific to a particular industry, because you don't want to help your direct competitors too much. But very little in-house software today is really industry-specific so there are still great improvements yet to be made.
For instance, when developing a new multi-million-dollar website, there is no harm in paying a couple of extra developers to attempt their own competing Plan B solution, just in case Plan A is a total failure.
Instant customer feedback
Open source projects have the benefit of direct feedback from users. Systems such as bugzilla and open mailing lists make it easy for customers to express their needs. That is the necessary first step to satisfying those needs. See the Structural Solutions section.
For instance, proprietary application server projects such as BEA and WebSphere seem deaf to the frustrations of their customers, but the open source JBoss project is happy to hear about those problems and avoid them in its own product.
Because they are so often distributed across the world, open source developers must use systems to communicate and to coordinate development instead of day-to-day personal contact in an office. Due to the complex nature of software development, these techniques should usually be used even when face-to-face contact is possible, because they allow more, and more constructive, communication.
At the very least they use email lists that are openly archived. Ideally they also use websites, bug-trackers, and source code control systems.
While many proprietary companies have software development "procedures", the public nature of open source development ensures that inappropriate procedures are replaced with procedures that actually increase quality. It is not enough to have a process or to believe that your process would work if it was followed - your process must work in reality.
Source code control - e.g. CVS
Open source projects use source code repositories, such as CVS. This is particularly necessary because the development teams are extremely distributed and tend to work in parallel. But it is necessary even in small development teams.
This also helps to organise the development of stable and unstable branches. See the APIs section.
Although many proprietary projects use source code repositories, most do not do so with appropriate organisation. In a typical open source project only a core group of developers can make changes directly. Patches from new, or less-involved, developers must first be reviewed and maybe modified. This process helps to nurture and integrate new developers into the team. This system doesn't just encourage teamwork and sharing of expertise - it demands it.
Bug tracking - e.g. bugzilla
Successful open source projects use a database of bugs which is available to all.
These can then generate constant self-writing status reports. For instance: GNOME's bug report. Everyone can see these reports. Both users and developers can see how many people are experiencing problems, and how significant these problems are. This creates increased pressure to prioritise and correct problems.
Bug-tracking system which only allow information to be entered by software developers rather than users are almost useless. Nothing should prevent someone from telling you that your software does not work. And neither should the right to view bugs be restricted to paying customers - you must increase the external pressure to increase quality. A well-run bug-tracker is an advertisment for your product's quality.
Easily buildable. Always buildable.
Open source projects can always be built, without compilation problems. And it's very easy to build them - either because they use well-known standard methods or because the build is documented. Because so few people work on any particular proprietary project it is very easy for the developers to accept inappropriately complex or undocumented build systems. They often believe that they will improve this later, but in reality, later never happens.
Difficult build systems encourage projects to hide from demands for quality, because a system that can not be built easily can not be evaluated continually. It also makes it very difficult to integrate new developers, and even wastes the time of the original developers.
In the worst cases, difficult or undocumented build systems hide the fact that source code does not build at all. Crises that are hidden are not solved.
For instance,
- mozilla is now much easier to build than the original Netscape source code.
- OpenOffice is now much easier to build than the original StarOffice source code (though not perfect).
- GNOME is built constantly and each module uses a consistent standard build system that can be automated.
Successful open source software has regular releases and does not wait for certain features to be finished before releasing. People must not wait longer for all 10 new features/bugfixes just because 1 of them is not finished, and developers must not wait for finished code to be tested. This is often called "release early, release often".
Most software projects consistently fail to accurately manage their feature-lists, or implementation times. Projects such as GNOME have been brave enough to recognise this and incorporate it into their release process. For instance, the GNOME community agrees on what future functionality would be desirable, and prioritizes its bugs, but accepts that the regular releases should just ship whatever has actually been achieved at that time. Of course the GNOME release process is more involved, with a schedule to allow coordination between teams, but this is the principle on which it is based.
In contrast, many proprietary managers choose to wait for ever, or until it is absurdly late. In general, customers would rather have something now than wait a year for everything. As long as there are frequent releases then the customer will get the functionality that he wants as soon as possible, and the greater number of releases increases the likelihood that a combination of modules exists which meets his needs. Again, see the APIs section to see how modules can be released individually and still work together.
Remember, if the customer doesn't understand this and says that a release without the requested functionality is unacceptable, you can just tell him that he doesn't have to use it if he doesn't want to. The customer is free to wait longer if he likes - you might even not tell him about it. But other customers should not have to wait also.
For instance, Java and Microsoft's development plaforms show what goes wrong when time-based releases are not used. The entire development platform is released together, so people have to wait a year or more between releases, not knowing whether their problems are solved until that time. When they discover that the problem is not fixed, they must keep hope alive for another year. They are forced to upgrade unrelated components at the same time, usually introducing new and unnecessary bugs. This internal memo describes the Java problem specifically.
Copyright © Murray Cumming.
