info
Morphium V5
Morphium
originated in 2012 and has had 4 major releases since then. Now it's time to update some more and tweak some internals. This update is a major one, so parts of the internals have been completly rewritten while keeping Morphium itself sourcecode compatible.
Targets of the rewrite
The aim is to cut off old "braids" and update the code. It should remain as compatible as possible so that migration is easy.
Furthermore, the complexity should be reduced, there are some features in Morphium
that are certainly rarely used or not at all, such as "optimistic locking" with the use of automatic versioning. These "features" are removed and the code is streamlined.
Furthermore, Morphium
should remain compatible with the older versions. That means the changes to Morphium
itself will be limited. The concepts will not change either, at most they will be tightened up a bit. So Morphium
is still the linchpin of the API and every write access will continue to run via Morphium
and every read access via Query
(exception: aggregation with the help of the aggregator). In short: the use of Morphium
essentially does not change.
JDK 11
The JDK used should also be updated during the refresh. Morphium
V4 stays on JDK1.8, from V5 we will support JDK11 and later. Of course, the code should also use the new features of JDK11.
Morphium Object Mapping
In current tests, the Morphium Object Mapper
is still one of the fastest and most feature-rich object mappers that we have found. Compared to 'Jackson', the higher performance and the features for handling lists and maps (handling 'Generics') have to be mentioned in particular - this makes the MorphiumObjectMapper
really indispensable.
During the rewrite, however, some features that are also reflected in the object mapping were removed, which simplifies the code and makes it easier to maintain.
Morphium Cache and WriteBuffer
Morphium
had declarative caching as part of the system from the start. This continues to be a key component. The cache implements the JCache
API and could therefore be used for other purposes as well. The other way around, you can also use a JCache in Morphium
for caching.
During the rewrite we will adapt the code to JDK11 and improve the structure of the code.
Morphium Messaging
Another key component is messaging.
Particular attention will be paid to messaging to improve stability and reduce load. Specifically, by efficiently using the Driver API and WireProtocol. Another change will be the connection handling in messaging: currently messaging uses Morphium
itself as a connector to the mongo. Since none of the features of Morphium
are necessary for messaging or are sometimes even more of a hindrance, the messaging will communicate directly with the MongoDB in V5
. In the current V5 implementation, the messaging uses a dedicated connection to the MongoDB for the push notifications (Watch
) and a dedicated connection pool for the connection to the mongo.
This means that the messaging is a little more detached from Morphium
, generates less load and can be configured separately. This is implemented and the tests show, that the performance increases whereas the load on mongo is reduced.
MorphiumConfig
This "thing" has also grown a little over the years and is currently hardly usable with the 100s of getters and setters. The settings have to be split up a bit and used in different categories. In particular, because Morphium
's own wire -protocol driver requires different settings than e.g. the InMemDriver
.
Focus on Community Edition
However, we only implement this for the Community Edition. This means that the enterprise features are not considered in the rewrite. It can also cause the drivers not to work properly with the Enterprise MongoDB version. The same currently applies to the Atlas MongoDB implementation, which will certainly be the first to be submitted. Support for the Enterprise Edition features may also be implemented in one of the following versions if required.
Rewrite bottom up
The rewrite is done from "below", i.e. first the Mongo driver (see below) was implemented, which implements the Wire protocol (currently only usable for MongoDB from V5! For older MongoDB versions please use Morphium
V4.x!).
The necessities and the structure of this driver is mainly determined by the MongoDB and propagates the architecture "up". On the way there, unused features are removed (e.g. auto-versioning), others are improved simply by adapting them to the new drivers.
In the end, a Morphium
should come out that is almost 100% source compatible with the old version, but much more modern, slimmer, more beautiful!
Morphium driver
Morphium
is more or less a replacement for the existing mongoDB driver with a lot more features and in some places significantly more security and / or performance.
Currently (i.e. in Morphium V4.x
) there are 2 drivers: MongoDriver
and InMemDriver
. The MongoDriver
in particular creates a bit of overhead because it is based on the official MongoDB driver, which unfortunately is now not just a driver, but also an object mapper. Many functions of the driver are not required at all, but some others are missing. So the aim is to introduce our own WireProtocol
[^1:WireProtocol is the protocol that MongoDB uses for communication] driver, which is optimized for the needs of morphium
. This can result in a significant performance advantage (currently between 10% and 20%).
2 drivers are implemented in V5:
SingleConnectDriver
- a driver that only keeps a single connection to the MongoDb, but monitors it and also implements a failover if necessary. This is used, for example, in the new messaging substructure (see below)PooledDriver
- a full featured cluster connector with connection pooling and automatic failover.
InMemory Driver
Strictly speaking, there is a third driver, the InMemoryDriver
, which makes it possible to run Morphium
with an "in memory" mongo. This is particularly useful for tests. We use this as a TestDB in some projects and it is also comparable to other databases implemented for test purposes.
The InMemory database should be (nearly) feature compatible to a real MongoDB installation. We're not 100% there yet, but we've reached a good 90%. Most features are already working, but some things take a little longer and might come later (such as "real" text indexes).
Why not the MongoDB driver
This question is asked more often and I would like to say a few words about it:
Features of the driver
The official MongoDB Java driver is not only a driver, but "embedded" in the driver is also an object mapper. This is probably super useful for many applications and completely sufficient.
Unfortunately not for us, because we want to configure some things rather than program them (declarative indices, etc.).
Two different ways of thinking collide. Which in the example of the MongoDB driver and Morphium
has repeatedly led to... discrepancies. Some changes to the driver API have been a lot of work in Morphium
.
The MorphiumDriver
project has been around for a few years. It was brought to life because we had problems with the official MongoDB driver, especially in the area of ​​failover. Also, some features of the driver are at least a little questionable in my opinion.
These features, especially the object mapping, make access via Morphium
a bit unwieldy. And some things happen unnecessarily twice. And that's exactly why we want to try to develop our own driver that does exactly what we need - and nothing more.
When accessing the Mongo in the stack trace, there are currently a good 20 calls within the MongoDB driver. With the new driver that's a maximum of 5!
new feature: InMemoryServer
Morphium
in version 5 got its own server, which is also based on the InMemory database. This is naturally intended exclusively for test purposes and does not offer any security or anything similar. However, it is always helpful for testing a (simple) in memory mongodb (with the limitations of the InMemoryDriver).
If you want to test it now:
The MorphiumServer
understands the following command line options:
-mt
or--maxThreads
: maximum number of threads in the server (= maximum number of connections)-mint
or--minThreads
: minimal number of threads (kept ready)-h
or--host
: which host does this server report back as (in the response to thehello
orisMaster
command)-p
or--port
: which port should be opened (all network IPs)
I would like to reiterate that this server is purely for testing purposes and is certainly not intended to be a replacement for any MongoDB installations.
migration
Migration should be relatively easy since Morphium
's API itself hasn't changed. However, the configuration has changed:
- the class names were removed. It makes no sense to include your own query implementation. At least we were not aware that anyone had done that. The same applies to the defined factories etc.
- The drivers are no longer referenced via the
ClassName
, now theDriverName
is specified. Valid drivers currently are:InMemoryDriver
,SingleConnectionDriver
,PooledDriver
,MongoDriver
. Unfortunately, the MongoDriver is currently not working and it is not 100% certain that it will remain part ofMorphium
- direct access to the driver - in case someone did it - is still possible, but the driver API has changed significantly. That's why you have to reckon with major renovation work here.
- the
optimistic locking
feature has been removed. Unfortunately, everyone who has used this feature currently has to find their own implementation
Current status
Currently, at the end of August 2022, Morphium V5
is available as an 'ALPHA' version in Maven Central and can be used for your own tests. However, some features are not yet 100% implemented. This is how Morphium works only with MongoDB versions 5.0 and later. Also, the transaction feature is currently not working.
plan
in the coming weeks these things will be addressed in this priority:
- Test stability / failover etc and adjust if necessary
- Implement auto retry read/write everywhere
- Finalise
InMemoryDriver
- add missing features, increase performance. - Rewrite MorphiumConfig and remove old settings, adapt new ones, possibly make them more modular.
- Implement transaction handling (at driver level, morphium has supported this for a long time)
- Complete or remove
MongoDriver
.