Vinod Valloppillil (VinodV)

Aug 11, 1998 – v1.00

Microsoft Confidential

Open Source Software (OSS) is a development process which promotes rapid creation and deployment of incremental features and bug fixes in an existing code / knowledge base. In recent years, corresponding to the growth of Internet, OSS projects have acquired the depth & complexity traditionally associated with commercial projects such as Operating Systems and mission critical servers.

Consequently, OSS poses a direct, short-term revenue and platform threat to Microsoft – particularly in server space. Additionally, the intrinsic parallelism and free idea exchange in OSS has benefits that are not replicable with our current licensing model and therefore present a long term developer mindshare threat.

However, other OSS process weaknesses provide an avenue for Microsoft to garner advantage in key feature areas such as architectural improvements (e.g. storage+), integration (e.g. schemas), ease-of-use, and organizational support.

Open Source Software (OSS) is software in which both source and binaries are distributed or accessible for a given product, usually for free. OSS is often mistaken for “shareware” or “freeware” but there are significant differences between these licensing models and the process around each product.

The broad categories of licensing include:

This paper focuses on Open Source Software (OSS). OSS is acutely different from the other forms of licensing (in particular “shareware”) in two very important respects:

Open source software has roots in the hobbyist and the scientific community and was typified by ad hoc exchange of source code by developers/users.

Internet Software

Free Software Foundation / GNU Project

The original FSF / GNU initiatives fell short of their original goal of creating a completely OSS Unix. They did, however, contribute several famous and widely disseminated applications and programming tools used today including:

CopyLeft Licensing

FSF/GNU software introduced the “copyleft” licensing scheme that not only made it illegal to hide source code from GNU software but also made it illegal to hide the source from work derived from GNU software. The document that described this license is known as the General Public License (GPL).

The second clause – open source code of derivative works – has been the most controversial (and, potentially the most successful) aspect of CopyLeft licensing.

Commercial software development processes are hallmarked by organization around economic goals. However, since money is often not the (primary) motivation behind Open Source Software, understanding the nature of the threat posed requires a deep understanding of the process and motivation of Open Source development teams.

In other words, to understand how to compete against OSS, we must target a process rather than a company.

On a practical level, this insight means we can expect Microsoft's propaganda machine to be directed against the process and culture of open source, rather than specific competitors. Brace for it... }

Some of the key attributes of Internet-driven OSS teams:

Communication – Internet Scale

Coordination of an OSS team is extremely dependent on Internet-native forms of collaboration. Typical methods employed run the full gamut of the Internet’s collaborative technologies:

OSS projects the size of Linux and Apache are only viable if a large enough community of highly skilled developers can be amassed to attack a problem. Consequently, there is direct correlation between the size of the project that OSS can tackle and the growth of the Internet.

Common Direction

In addition to the communications medium, another set of factors implicitly coordinate the direction of the team.

Common Goals

Common goals are the equivalent of vision statements which permeate the distributed decision making for the entire development team. A single, clear directive (e.g. “recreate UNIX”) is far more efficiently communicated and acted upon by a group than multiple, intangible ones (e.g. “make a good operating system”).

Common Precedents

Precedence is potentially the most important factor in explaining the rapid and cohesive growth of massive OSS projects such as the Linux Operating System. Because the entire Linux community has years of shared experience dealing with many other forms of UNIX, they are easily able to discern – in a non-confrontational manner – what worked and what didn’t.

There weren’t arguments about the command syntax to use in the text editor – everyone already used “vi” and the developers simply parcelled out chunks of the command namespace to develop.

Accordingly, it suggest that Microsoft's response to open source can be wrong-footed by emphasizing innovation in both our actions and the way we represent what we're doing to the rest of the world. }

NatBro points out that the need for a commonly accepted skillset as a pre-requisite for OSS development. This point is closely related to the common precedents phenomena. From his email:

The Cathedral and the Bazaar

Raymond dissected his OSS project in order to derive rules-of-thumb which could be exploited by other OSS projects in the future. Some of Raymond’s rules include:

Every good work of software starts by scratching a developer's personal itch

Good programmers know what to write. Great ones know what to rewrite (and reuse).

``Plan to throw one away; you will, anyhow.''

Treating your users as co-developers is your least-hassle route to rapid code improvement and effective debugging.

Release early. Release often. And listen to your customers.

Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.

Once a component framework has been established (e.g. key API’s & structures defined), OSS projects such as Linux utilize multiple small teams of individuals independently solving particular problems.

Because the developers are typically hobbyists, the ability to ‘fund’ multiple, competing efforts is not an issue and the OSS process benefits from the ability to pick the best potential implementation out of the many produced.

Note, that this is very dependent on:

The core argument advanced by Eric Raymond is that unlike other aspects of software development, code debugging is an activity whose efficiency improves nearly linearly with the number of individuals tasked with the project. There are little/no management or coordination costs associated with debugging a piece of open source code – this is the key ‘break’ in Brooks’ laws for OSS.

Raymond includes Linus Torvald’s description of the Linux debugging process:

Put alternately:

One advantage of parallel debugging is that bugs and their fixes are found / propagated much faster than in traditional processes. For example, when the TearDrop IP attack was first posted to the web, less than 24 hours passed before the Linux community had a working fix available for download.

“Impulse Debugging”

An extension to parallel debugging that I’ll add to Raymond’s hypothesis is “impulsive debugging”. In the case of the Linux OS, implicit to the act of installing the OS is the act of installing the debugging/development environment. Consequently, it’s highly likely that if a particular user/developer comes across a bug in another individual’s component – and especially if that bug is “shallow” – that user can very quickly patch the code and, via internet collaboration technologies, propagate that patch very quickly back to the code maintainer.

Put another way, OSS processes have a very low entry barrier to the debugging process due to the common development/debugging methodology derived from the GNU tools.

Any large scale development process will encounter conflicts which must be resolved. Often resolution is an arbitrary decision in order to further progress the project. In commercial teams, the corporate hierarchy + performance review structure solves this problem – How do OSS teams resolve them?

In the case of Linux, Linus Torvalds is the undisputed ‘leader’ of the project. He’s delegated large components (e.g. networking, device drivers, etc.) to several of his trusted “lieutenants’ who further de-facto delegate to a handful of “area” owners (e.g. LAN drivers).

Other organizations are described by Eric Raymond: (http://earthspace.net/~esr/writings/homesteading/homesteading-15.html):

This section provides an overview of some of the key reasons OSS developers seek to contribute to OSS projects.

Solving the Problem at Hand

This is basically a rephrasing of Raymond’s first rule of thumb – “Every good work of software starts by scratching a developer’s personal itch”.

Many OSS projects – such as Apache -- started as a small team of developers setting out to solve an immediate problem at hand. Subsequent improvements of the code often stem from individuals applying the code to their own scenarios (e.g. discovering that there is no device driver for a particular NIC, etc.)

Education

The Linux kernel grew out of an educational project at the University of Helsinki. Similarly, many of the components of Linux / GNU system (X windows GUI, shell utilities, clustering, networking, etc.) were extended by individuals at educational institutions.

Ego Gratification

The most ethereal, and perhaps most profound motivation presented by the OSS development community is pure ego gratification.

In “The Cathedral and the Bazaar”, Eric S. Raymond cites:

And, of course, “you aren’t a hacker until someone else calls you hacker”

Homesteading on the Noosphere

A second paper published by Raymond – “Homesteading on the Noosphere” (http://sagan.earthspace.net/~esr/writings/homesteading/), discusses the difference between economically motivated exchange (e.g. commercial software development for money) and “gift exchange” (e.g. OSS for glory).

“Homesteading” is acquiring property by being the first to ‘discover’ it or by being the most recent to make a significant contribution to it. The “Noosphere” is loosely defined as the “space of all work”. Therefore, Raymond posits, the OSS hacker motivation is to lay a claim to the largest area in the body of work. In other words, take credit for the biggest piece of the prize.

From “Homesteading on the Noosphere”:

Altruism

This is a controversial motivation and I’m inclined to believe that at some level, Altruism ‘degenerates’ into a form of the Ego Gratification argument advanced by Raymond.

One smaller motivation which, in part, stems from altruism is Microsoft-bashing.

A key threat in any large development team – and one that is particularly exacerbated by the process chaos of an internet-scale development team -- is the risk of code-forking.

Code forking occurs when over normal push-and-pull of a development project, multiple, inconsistent versions of the project’s code base evolve.

In the commercial world, for example, the strong, singular management of the Windows NT codebase is considered to be one of it’s greatest advantages over the ‘forked’ codebase found in commercial UNIX implementations (SCO, Solaris, IRIX, HP-UX, etc.).

Forking in OSS – BSD Unix

Within OSS space, BSD Unix is the best example of forked code. The original BSD UNIX was an attempt by U-Cal Berkeley to create a royalty-free version of the UNIX operating system for teaching purposes. However, Berkeley put severe restrictions on non-academic uses of the codebase.

In order to create a fully free version of BSD UNIX, an ad hoc (but closed) team of developers created FreeBSD. Other developers at odds with the FreeBSD team for one reason or another splintered the OS to create other variations (OpenBSD, NetBSD, BSDI).

There are two dominant factors which led to the forking of the BSD tree:

Both of these motivations create a situation where developers may try to force a fork in the code and collect royalties (monetary, or ego) at the expense of the collective BSD society.

(Lack of) Forking in Linux

In contrast to the BSD example, the Linux kernel code base hasn’t forked. Some of the reasons why the integrity of the Linux codebase has been maintained include:

What are the core strengths of OSS products that Microsoft needs to be concerned with?

Like our Operating System business, OSS ecosystems have several exponential attributes:

Long-Term Credibility Defined

Long term credibility exists if there is no way you can be driven out of business in the near term. This forces change in how competitors deal with you.

For example, Airbus Industries garnered initial long term credibility from explicit government support. Consequently, when bidding for an airline contract, Boeing would be more likely to accept short-term, non-economic returns when bidding against Lockheed than when bidding against Airbus.

OSS is Long-Term Credible

OSS systems are considered credible because the source code is available from potentially millions of places and individuals.

This could be both good and bad news. The good news is that Microsoft would give up attack marketing, a weapon which in the past has been much more powerful than its distinctly inferior technology. The bad news is that, against us, giving it up would actually be better strategy; they wouldn't be wasting energy any more and might actually evolve some effective response. }

The likelihood that Apache will cease to exist is orders of magnitudes lower than the likelihood that WordPerfect, for example, will disappear. The disappearance of Apache is not tied to the disappearance of binaries (which are affected by purchasing shifts, etc.) but rather to the disappearance of source code and the knowledge base.

Inversely stated, customers know that Apache will be around 5 years from now -- provided there exists some minimal sustained interested from its user/development community.

One Apache customer, in discussing his rationale for running his e-commerce site on OSS stated, “because it’s open source, I can assign one or two developers to it and maintain it myself indefinitely. “

Lack of Code-Forking Compounds Long-Term Credibility

The GPL and its aversion to code forking reassures customers that they aren’t riding an evolutionary ‘dead-end’ by subscribing to a particular commercial version of Linux.

Linux and other OSS advocates are making a progressively more credible argument that OSS software is at least as robust – if not more – than commercial alternatives. The Internet provides an ideal, high-visibility showcase for the OSS world.

Maybe there's a message about the underlying products here? }

In particular, larger, more savvy, organizations who rely on OSS for business operations (e.g. ISPs) are comforted by the fact that they can potentially fix a work-stopping bug independent of a commercial provider’s schedule (for example, UUNET was able to obtain, compile, and apply the teardrop attack patch to their deployed Linux boxes within 24 hours of the first public attack)

Alternatively stated, “developer resources are essentially free in OSS”. Because the pool of potential developers is massive, it is economically viable to simultaneously investigate multiple solutions / versions to a problem and chose the best solution in the end.

For example, the Linux TCP/IP stack was probably rewritten 3 times. Assembly code components in particular have been continuously hand tuned and refined.

OSS’s API evangelization / developer education is basically providing the developer with the underlying code. Whereas evangelization of API’s in a closed source model basically defaults to trust, OSS API evangelization lets the developer make up his own mind.

On the other hand, they may be right. We in the open-source community can't afford to dismiss that possibility. I think we can meet it by developing high-quality documentation. In this way, `trust' in name authors (or in publishers of good repute such as O'Reilly or Addison-Wesley) can substitute for `trust' in an API-defining organization.) }

Strongly componentized OSS projects are able to release subcomponents as soon as the developer has finished his code. Consequently, OSS projects rev quickly & frequently.

The weaknesses in OSS projects fall into 3 primary buckets:

The biggest roadblock for OSS projects is dealing with exponential growth of management costs as a project is scaled up in terms of rate of innovation and size. This implies a limit to the rate at which an OSS project can innovate.

Starting an OSS project is difficult

From Eric Raymond:

Raymond ‘s argument can be extended to the difficulty in starting/sustaining a project if there are no clear precedent / goal (or too many goals) for the project.

Bazaar Credibility

Obviously, there are far more fragments of source code on the Internet than there are OSS communities. What separates “dead source code” from a thriving bazaar?

I’ll posit that some of the key criteria that must exist for a bazaar to be credible include:

Post-Parity Development

When describing this problem to JimAll, he provided the perfect analogy of “chasing tail lights”. The easiest way to get coordinated behavior from a large, semi-organized mob is to point them at a known target. Having the taillights provides concreteness to a fuzzy vision. In such situations, having a taillight to follow is a proxy for having strong central leadership.

This is possibly the single most interesting hurdle to face the Linux community now that they’ve achieved parity with the state of the art in UNIX in many respects.

Another interesting thing to observe in the near future of OSS is how well the team is able to tackle the “unsexy” work necessary to bring a commercial grade product to life.

In the operating systems space, this includes small, essential functions such as power management, suspend/resume, management infrastructure, UI niceties, deep Unicode support, etc.

For Apache, this may mean novice-administrator functionality such as wizards.

Integrative/Architectural work

Integrative work across modules is the biggest cost encountered by OSS teams. An email memo from Nathan Myrhvold on 5/98, points out that of all the aspects of software development, integration work is most subject to Brooks’ laws.

Up till now, Linux has greatly benefited from the integration / componentization model pushed by previous UNIX’s. Additionally, the organization of Apache was simplified by the relatively simple, fault tolerant specifications of the HTTP protocol and UNIX server application design.

We can only hope Microsoft continues to believe this, because it would hinder their response. Much will depend on how they interpret innovations such as (for example) the SMPization of the Linux kernel. }

These are weaknesses intrinsic to OSS’s design/feedback methodology.

Iterative Cost

One of the key’s to the OSS process is having many more iterations than commercial software (Linux was known to rev it’s kernel more than once a day!). However, commercial customers tell us they want fewer revs, not more.

The Linux OS is not developed for end users but rather, for other hackers. Similarly, the Apache web server is implicitly targetted at the largest, most savvy site operators, not the departmental intranet server.

The key thread here is that because OSS doesn’t have an explicit marketing / customer feedback component, wishlists – and consequently feature development -- are dominated by the most technically savvy users.

The interesting trend to observe here will be the effect that commercial OSS providers (such as RedHat in Linux space, C2Net in Apache space) will have on the feedback cycle.

How can OSS provide the service that consumers expect from software providers?

Support Model

Product support is typically the first issue prospective consumers of OSS packages worry about and is the primary feature that commercial redistributors tout.

For the short-medium run, this factor alone will relegate OSS products to the top tiers of the user community.

Strategic Futures

What does it mean for the Linux community to “sign up” to help build the Corporate Digital Nervous System? How can Linux guarantee backward compatibility with apps written to previous API’s? Who do you sue if the next version of Linux breaks some commitment? How does Linux make a strategic alliance with some other entity?

In the last 2 years, OSS has taken another twist with the emergence of companies that sell OSS software, and more importantly, hiring full-time developers to improve the code base. What’s the business model that justifies these salaries?

In many cases, the answers to these questions are similar to “why should I submit my protocol/app/API to a standards body?”

The vendor of OSS-ware provides sales, support, and integration to the customer. Effectively, this transforms the OSS-ware vendor from a package goods manufacturer into a services provider.

The Loss Leader OSS business model can be used for two purposes:

This is very closely related to the loss leader business model. However, instead of trying to get marginal service returns by massively growing the market, these businesses increase returns in their part of the value chain by commoditizing downstream suppliers.

The best examples of this currently are the thin server vendors such as Whistle Communications, and Cobalt Micro who are actively funding developers in SAMBA and Linux respectively.

Both Whistle and Cobalt generate their revenue on hardware volume. Consequently, funding OSS enables them to avoid today’s PC market where a “tax” must be paid to the OS vendor (NT Server retail price is $800 whereas Cobalt’s target MSRP is around $1000).

The earliest Apache developers were employed by cash-strapped ISPs and ICPs.

Another, more recent example is IBM’s deal with Apache. By declaring the HTTP server a commodity, IBM hopes to concentrate returns in the more technically arcane application services it bundles with it’s Apache distribution (as well as hope to reach Apache’s tremendous market share).

One of the exponential qualities of OSS – successful OSS projects swallow less successful ones in their space – implies a pre-emption business model where by investing directly in OSS today, they can pre-empt / eliminate competitive projects later – especially if the project requires API evangelization. This is tantamount to seizing a first mover advantage in OSS.

In addition, the developer scale, iteration rate, and reliability advantages of the OSS process are a blessing to small startups who typically can’t afford a large in–house development staff.

Examples of startups in this space include SendMail.com (making a commercially supported version of the sendmail mail transfer agent) and C2Net (makes commercial and encrypted Apache)

Notice, that no case of a successful startup originating an OSS project has been observed. In both of these cases, the OSS project existed before the startup was formed.

Sun Microsystem’s has recently announced that its “JINI” project will be provided via a form of OSS and may represent an application of the pre-emption doctrine.

The next several sections analyze the most prominent OSS projects including Linux, Apache, and now, Netscape’s OSS browser.

A second memo titled “Linux OS Competitive Analysis” provides an in-depth review of the Linux OS. Here, I provide a top-level summary of my findings in Linux.

Linux (pronounced “LYNN-ucks”) is the #1 market share Open Source OS on the Internet. Linux is derives strongly from the 25+ years of lessons learned on the UNIX operating system.

Top-Level Features:

Like other Open Source Software (OSS) products, the real key to Linux isn’t the static version of the product but rather the process around it. This process lends credibility and an air of future-safeness to customer Linux investments.

The primary threat Microsoft faces from Linux is against NT Server.

Linux’s future strength against NT server (and other UNIXes) is fed by several key factors:

Linux is unlikely to be a threat in the medium-long term on the desktop for several reasons:

In addition to the attacking the general weaknesses of OSS projects (e.g. Integrative / Architectural costs), some specific attacks on Linux are:

In an attempt to renew it’s credibility in the browser space, Netscape has recently released and is attempting to create an OSS community around it’s Mozilla source code.

Netscape’s organization and licensing model is loosely based on the Linux community & GPL with a few differences. First, Mozilla and Netscape Communicator are 2 codebases with Netscape’s engineers providing synchronization.

Capitalize on Anti-MSFT Sentiment in the OSS Community

Relative to other OSS projects, Mozilla is considered to be one of the most direct, near-term attacks on the Microsoft establishment. This factor alone is probably a key galvanizing factor in motivating developers towards the Mozilla codebase.

New credibility

Scratch a big itch

The browser is widely used / disseminated. Consequently, the pool of people who may be willing to solve “an immediate problem at hand” and/or fix a bug may be quite high.

Post parity development

Mozilla is already at close to parity with IE4/5. Consequently, there no strong example to chase to help implicitly coordinate the development team.

Netscape has assigned some of their top developers towards the full time task of managing the Mozilla codebase and it will be interesting to see how this helps (if at all) the ability of Mozilla to push on new ground.

Small Noosphere

An interesting weakness is the size of the remaining “Noosphere” for the OSS browser.

Integration Cost

Potentially the single biggest detriment to the Mozilla effort is the level of integration that customers expect from features in a browser. As stated earlier, integration development / testing is NOT a parallelizable activity and therefore is hurt by the OSS process.

In particular, much of the new work for IE5+ is not just integrating components within the browser but continuing integration within the OS. This will be exceptionally painful to compete aga inst.

The contention therefore, is that unlike the Apache and Linux projects which, for now, are quite successful, Netscape’s Mozilla effort will:

Internal mirrors of the Mozilla mailing lists can be found on http://egg.Microsoft.com/wilma/lists

Leadership:

Dispute resolution:

Market Share!

Apache far and away has #1 web site share on the Internet today. Possession of the lion’s share of the market provides extremely powerful control over the market’s evolution.

In particular, Apache’s market share in web server space presents the following competitive hurdles:

3^rd Party Support

The number of tools / modules / plug-ins available for Apache has been growing at an increasing rate.

Performance

In the short run, IIS soundly beats Apache on SPECweb. Moving further, as IIS moves into kernel and takes advantage deeper integration with the NT, this lead is expected to increase further.

Apache, by contrast, is saddled with the requirement to create portable code for all of its OS environments.

HTTP Protocol Complexity & Application services

Part of the reason that Apache was able to get a foothold and take off was because the HTTP protocol is so simple. As more and more features become layered on top of the humble web server (e.g. multi-server transaction support, POD, etc.) it will be interesting to see how the Apache team will be able to keep up.

ASP support, for example is a key driver for IIS in corporate intranets.

Recently, IBM announced it’s support for the Apache codebase in its WebSphere application server. The actual result of the press furor is still unclear however:

Some other OSS projects:

In general, a lot more thought/discussion needs to put into Microsoft’s response to the OSS phenomena. The goal of this document is education and analysis of the OSS process, consequently in this section, I present only a very superficial list of options and concerns.

Where is Microsoft most likely to feel the “pinch” of OSS projects in the near future?

Server vs. Client

The server is more vulnerable to OSS products than the client. Reasons for this include:

The ability of the OSS process to collect and harness the collective IQ of thousands of individuals across the Internet is simply amazing. More importantly, OSS evangelization scales with the size of the Internet much faster than our own evangelization efforts appear to scale.

How can Microsoft capture some of the rabid developer mindshare being focused on OSS products?

Some initial ideas include:

What can Microsoft learn from the OSS example? More specifically: How can we recreate the OSS development environment internally? Different reviewers of this paper have consistently pointed out that internally, we should view Microsoft as an idealized OSS community but, for various reasons do not:

The trick of course, is to capture these benefits without incurring the costs of the OSS process. These costs are typically the reasons such barriers were erected in the first place:

Supporting a platform & development community requires a lot of service infrastructure which OSS can’t provide. This includes PDC’s, MSDN, ADCU, ISVs, IHVs, etc.

The OSS communities “MSDN” equivalent, of course, is a loose confederation of web sites with API docs of varying quality. MS has an opportunty to really exploit the web for developer evangelization.

Generally, Microsoft wins by attacking the core weaknesses of OSS projects.

De-commoditize protocols & applications

OSS projects have been able to gain a foothold in many server applications because of the wide utility of highly commoditized, simple protocols. By extending these protocols and developing new protocols, we can deny OSS projects entry into the market.

David Stutz makes a very good point: in competing with Microsoft’s level of desktop integration, “commodity protocols actually become the means of integration” for OSS projects. There is a large amount of IQ being expended in various IETF working groups which are quickly creating the architectural model for integration for these OSS projects.

Once again, open-source advocates' best response is to point out to customers that when things are ``de-commoditized'', vendors gain and customers lose. }

Some examples of Microsoft initiatives which are extending commodity protocols include:

Make Integration Compelling – Especially on the server

The rise of specialty servers is a particularly potent and dire long term threat that directly affects our revenue streams. One of the keys to combating this threat is to create integrative scenarios that are valuable on the server platform. David Stutz points out:

Organizational Credibility

Many people provided, datapoints, proofreading, thoughtful email, and analysis on both this paper and the Linux analysis: