Requiescat in Pace: SpringSource dm Server

SpringSource recently gave ownership of the SpringSource dm Server project to the Eclipse foundation, as the Virgo Dynamic Enterprise Application Platform, along with the Spring Dynamic Modules project, which is now the Gemini Enterprise Modules project. According to SpringSource, the dm Server did not achieve the market success that they wanted, so while they are continuing to work on the now-external projects (they retain committer status, for example), they are no longer officially in that particular business.

In one sense, this is yet another software project shot out from under me. (Hey, I can add it to the long list that makes up most of my resume!) In another sense, however, it actually does not matter all that much: we were not paying SpringSource for support, so the lack of that does not hurt us any. As long as the Eclipse projects are live, I have no qualms about continuing to use the dm Server, no matter what name is applied to it. Further, the ways we have been using the dm Server have mostly followed the OSGi and related standards. With the addition of the Web Application Specification and the Blueprint Services to the OSGi Service Platform Enterprise Specification, I think we are actually in pretty good shape.

However, this is a good opportunity to look at the dm Server, and the entire use of OSGi-based enterprise servers. At the risk of recreating the Celestial Emporium of Benevolent Knowledge, my retrospective here looks at three areas:

• Attributes of our particular design.
• Things OSGi brings to the table.
• Things the dm Server does well and things it does badly.

Home is where you hang your hat

Our system supports development, maintenance, and production use of a fair number of mostly small, relatively simple Java-based web applications. Some of the web applications represent custom development, while others are either off-the-shelf applications or locally developed "legacy" applications. (I dislike the label "legacy" intensely, but in this case it seems to be applicable; these applications represent traditional Java web applications that do not see enough continuing development to customize to use OSGi or our framework.) The custom-developed applications cross the gamut from JRuby on Rails property tracking to Flex personnel directory to Google Web Toolkit-based portlets.

To support the custom-developed applications, we have a library of framework code made up of database-backed configuration information, support for connections to remote systems incuding databases, LDAP, and SAP, and a variety of authentication and authorization services. (The importance of keeping configuration information in a live state has been touched upon by a co-worker.) In addition, we have a collection of the ususal Java suspects: Apache commons, XML processing, and so on.

The primary constraints have been ease of development and ease of production maintenance. None of our applications have seen enough use so far to make scaling or efficiency an issue. Although latency is always important for web applications, it has been less a framework problem than a constraint on the individual applications. However, given the number of applications, developers, and support personnel, development and production support are serious problems.

Ideally, each of the applications would be isolated from the others, uncoupling the applications from each other and allowing their framework and library components to evolve separately. Unfortunately, institutional limits on the number of IP addresses available make that problematic. Also, having n separate applications would mean having n processes, n copies of the framework and libraries, and roughly n times the memory footprint of a single application server. Finally, it means nearly n times the management overhead, which, again due to institutional constraints, is considerable. Which argument is overwhelming depends on the phase of the moon.

Using a single application server for multiple applications introduces the opposite problem: without some outside assistance, the applications, framework, and libraries have to be synchronized, forcing all of the applications to upgrade in lock-step for example. The cost of regression testing otherwise-unchanged applications limits flexibility for ongoing work. This is where OSGi comes in to the picture. In its simplest features, OSGi allows multiple versions of a jar file to coexist, handling an application's classpath dynamically based on exported and imported Java packages. This modularity also allows internal-use-only packages to be kept from application's environment.

The primary additional feature that the SpringSource dm Server adds to OSGi is the ability to deploy war files as full-fledged elements of the OSGi environment. It does so roughly using OSGi's extender pattern, adding the extra metadata required by OSGi to the war file dynamically as it is deployed; if the war file already has the metadata, it can make use of packages and services from OSGi.

In practice, traditional web applications are self-contained, holding all of their libraries and configuration information within the archive file. They can be deployed normally to the SpringSource server. Custom-developed applications use OSGi's mechanisms to import packages and use OSGi services; they explicitly describe which versions of library packages they require, and further which services they use. Upgrading a library jar in a backwards-compatible way (a bug-fix, for example) is as simple as deploying the jar to the server and rewiring the applications (and other bundles) which use it. Fixing a service is even simpler: references to services are acquired dynamically, so simply deploying the service-provider bundle allows relevent applications to use it. Finally, applications which cannot use a non-backwards compatible change will not even be able to see the new packages because of the constraints on imports and exports.

Serves 12, or 3 ranch hands

In most cases, the use of OSGi in our enterprise environment follows the typical patterns for OSGi applications. However, there are some differences between this, enterprise, use of OSGi and the more typical embedded and application use.

For one thing, there is always the question of architecture for OSGi bundles providing services: Should the API interface of the service be in the same bundle as the class implementing the service. As is implied by a previous paragraph, our answer to that question is that the API should be separated from the implementation, allowing the services to be updated without replacing the much-slower-changing API.

The next issue is version management. Given a package exported with verison x.y.z, what range should an importing bundle specify? For most cases, the right answer is [x.y.z, x+1.0.0); any version up to but not including the next major version. (A one-sentence primer on OSGi versions: major version numbers should be incremented on backwards-incompatible changes, minor version numbers on backwards-compatible API changes, and micro version numbers on completely compatible changes.) Keep in mind that the behavior of Java interfaces is different depending on whether the interface is implemented by the user or simply used. Any change to an implemented interface is incompatible, while interface use is more lenient.

The situation is slightly different for bundles importing API packages in order to provide services. In this case, the proper answer seems to be to import the API package with a version range of [x.y.z, x.y+1.0). With this range, any version of the package is valid up to but not including the next minor change. Normally, any API change will require either a major or minor increment, so this import will be restricted to the API that the service bundle implements, while micro increments of the API package allow classes in the API package to be changed as needed. On the other hand, if the API package is imported using the major increment policy described above, then implementation bundles for versions 2.8, 2.9, and 2.10 will all import the 2.10 API and provide their services against that API (which is perfectly legitimate from the version management standpoint; 2.8 is compatible with 2.10, minus some added features). As a result, applications expecting the 2.9 version will be service-less.

One piece of OSGi best-practice is not well-founded for this enterprise environment: the advice for a bundle to import the packages it is exporting. This suggestion is valid for normal OSGi applications, where a single class-space throughout the single application is desirable. In our case, however, we actively want separate class-spaces. Specifically, if we have both version 1.1.3 and 1.1.4 of Apache Weasel, we potentially want both versions available, not just 1.4 re-exported by the 1.3 bundle. As a result, Weasel's bundle needs to, at best, import its packages at its specific version number; at worst, it should not import them at all.

Finally, the "uses" clause of package exports needs extreme care. Peter Kriens' excellent bnd tool generates the uses clause automatically, but the clause it generates requires all of the packages possibly used by the exported package. This is a safe choice for the tool, but, like concurrency, the safe choice may not always be the best one. In our case, the automatically generated uses clause is frequently overconstraining. For one example, it includes the logging package which is only used incidentally by the implementation of classes in the exported package. (Okay, that is a poor example; there is only one logging package in our framework. However, the problem is that the internal implementation detail is escaping into the bundles' interface to the system.) Another example I recently encountered involved our security service API and a package of utility servlet filters. An application I was adding required one of the filters and version 2.12 of the API, while the bundle the filter package lives in was wired to 2.11 on a specific server and had a uses clause for part of the security API. The worst part of the situation was that the specific filter I was using did not use the security API; that was required (and its interface would indeed need the uses clause) by another filter class in the same package. In short, I was an idiot when I created that package and being hoisted with my own petard when deploying the application. In any case, I have not tracked down all of the examples where this unnecessary constraint could be a problem.

Dammit, Jim, I'm a doctor, not a bricklayer

Throughout our adventure with the SpringSource dm Server, the dm Server has performed very well. Its web services are based on Apache Tomcat, which is exceptionally stable, and it has mostly shown only the normal collection of glitches. There were, however, a few significant pain points. (Keep in mind, we have only used the 1.0 versions of the server; it is possible, even likely, that some or all of these have been addressed with 2.0.)

The first is that not all the normal OSGi actions work correctly in the SpringSource server. Specifically, I have had problems with rewiring installed bundles using the Equinox console's "refresh" command. When trying to rewire an application bundle the update caused the application to stop working.

The second is that, unlike web applications deployed to Tomcat or OSGi bundles deployed in most OSGi systems, the SpringSource server redeploys bundles and applications when it is restarted. As a result, changes which would normally be persistent, such as OSGi's Config Admin configurations or file system changes by web applications, are overwritten with the default state of applications and bundles.

Now, that last paragraph does not seem to be strictly true. Sometimes, for example, bundle wiring does not get updated on restarts, or is only partially updated. Multiple restarts seem to be required to get everything to stabilize.

The way bundles are started in the dm Server is a third, larger problem. Now, I have not confirmed this in the code, but it seems that a single thread performs bundle deployment and startup when the system is initializing. An application or framework bundle using SpringSource's Dynamic Modules service model with a hard requirement for some specific service (or which uses the service during its start-up) will block the initialization thread waiting for the service if the service is not already available. Eventually the thread will time out, cause the application or framework bundle to fail to deploy, and continue starting bundles. Possibly including the bundle which supplies the service the application was looking for. (This problem is not strictly limited to applications using the DM service model; any application with a hard requirement may fail to re-deploy. However, we have most often seen it with DM bundles that do not mark the service as "optional".)

The biggest complaint, however, is that our Java developers are not used to the modularity supported by OSGi. The idea of explicitly managing imports and exports seems to them to be unnecessary work. Certainly, Java development tools such as Eclipse actively conspire to hide modularity decisions. For example, Eclipse makes importing a class as easy as typing part of the name and hitting control-space. This easy functionality completely obscures even Java's normal import requirements. For example, probably the most common Base 64 encoder in Java code is the one provided by Sun's Java implementation, the one in an internal-use-only sun package. The use of this class is everywhere, even though it is undocumented and generates a warning from the compiler.

SpringSource's tools do help, as presumably do other features of other development environments. However, those introduce another three-step headache: convincing developers to install and use the tools, subsequently explaining why the developer does not see all of the classes they expect to see and what to do about it, and then continually listening to the whining about the multi-step complexity.

For more details about developing with OSGi, Check out A Year With OSGi at Soup In A Deli.

I laughed, I cried, it changed my life

Overall, pain aside, our use of the SpringSource dm server is moderately successful. Although I have not discussed it, the discipline of OSGi-enforced modularity does improve code quality, for those developers which take advantage of it. That modularity and OSGi's services have made system and service upgrades much less tooth-grinding. The servers themselves are exceptionally stable.

It is unfortunate that the dm Server has not seen the market penetration that SpringSource envisioned. In our niche, however, it seems to have filled a set of requirements that nothing else could. Given the standardization efforts on the features we use and need, and our institutional situation, I cannot see us needing to move to any other approach.

The King is dead! Long live the King!

[Edited to add links and fix some pronoun references.]

Comments

Thanks for writing up your experience with dm Server. Some, but not all, of the problems you encountered are indeed solved in v2. For instance, plans allow you to deploy groups of bundles and opt out of scoping and atomicity, which will make manipulating bundles individually more straightforward. There are still some weakness in the area of update, but I expect that to improve over time in Virgo.

As for the tooling, things are gradually improving. For example, IntelliJ recently shipped support for dm Server, which gives another choice of IDE.

If you eventually upgrade to v2 or Virgo (v2.1 and beyond), please do consider raising bugzillas for limitations that are still present. We depend on feedback from users to refine the behaviour.

Glyn
2010-06-06

Glyn, thanks for your comments and for your extraordinary work! I am playing with v2 now, and intend to try to get more involved with Virgo.

Tommy McGuire
2010-06-11

Copyright © 2007-2017 Tommy M. McGuire