Morphium based blogsoftware

published : 2019-07-29 changed: 2019-07-30

category: Computer --> programming --> MongoDB --> morphium

tags: jblog


Morphium based blog software Jblog

Some time ago I "dared" to leap away from Wordpress and reported about it (https://caluga.de/v/2017/5/19/jblog_java_blogging_software). Now it's been over 2 years (: astonished :) and there it is time to draw a conclusion and to report a little how it went.

Jblog - the good the bad the ugly

The good anticipation: the server was not hacked. The hits that were aimed at any PHP gaps or so are all ineffective "bounced" - that's what I wanted to achieve. The update topic is also solved: there was no: smirk: I took care of the further development myself. However, we already had a few bugs, and some are a bit annoying. Also, some things have turned out to be less practical. The quite elaborate versioning of the individual blog posts is actually unnecessary and complicates everything only.

  But otherwise...

Technology

What is currently behind Jblog:

  • Morphium V4.0.6
  • Spring boat
  • Apache Freemarker
  • Bootstrap 4
  • JDK11
  • MongoDB 4.0

yes, that was it already, has become quite clear. That The application is currently running as a Spring Boot app behind a nginx.  

features

I knit my blog as ich wanted it. So very much in my way of blogging, which may be the reason why nobody forked on Github and I changed the project to privat. : Sob:

multilingualism

Jblog is consistently bilingual. Theoretically, more would be possible, but I wanted to focus on German and English. That every element on the website is translated, all blog entries must be recorded in 2 languages.

For this a MessageSource is used which reads the data from Mongo. The nice: if a translation is not found, a link to the translation mask will be issued. That when I run Jblog on an empty translation database, I get a screen full of links that take me to the translation tool. Gradually this will be corrected.

By using Morphium to access the Mongo, these translations can also be cached (just write the @Cache annotation to the class).

The nice thing is that the cache, if wanted, is cluster-aware. That I can run multiple instances of Jblog and access through a loadbalancer. And these would keep their caches in sync across the Mongo, i. there are no dirty reads even with round-robin attitude!

Markdown

Markdown is indeed der hit lately to enter text. I think that's pretty cool too, especially since I type very well and the grip on the mouse always slows me down. That's why it's so easy to type everything. (and who like me used to do word processing with LaTeX knows this: smirk :.)

Jblog is completely designed for Markdown. That You enter his blogs in Markdown and then see the preview right away. I have also made some extensions, especially to simply emojis (: confused:: smile:: sweat :) integrate or embed a nicely formatted Java code block.

I also implemented the embedding of pictures.

I used a Markdown implementation called [Markingbird] (https://github.com/pmattos/Markingbird) and adjusted it a bit. So I've built something to embed the emojis easier and easier to view images.

Each blog post is written in Markdown and will be rendered on first access. And the result is cached, of course: smirk:

When typing, I have a live preview and can see exactly how the post will look at the end.

Twitter connection

new articles are published via Twitter. Some things have been taken into consideration: specify the time of the tweet, only for new articles etc.

And it is tweeted in 2 languages, German and English

multiuser capability

Yes, you can actually use Jblog with several users. Although I have a few friends and acquaintances, the time here from time to time pure snow and possibly what to post, but otherwise ... Is so a very little used feature, which is why I have then deactivated.

At the moment you can not register, the registration process is De-Enabled. If you want to leave a comment, you can do that via Disqus.

Implementation

The implementation has changed a bit in the last 2 years. Launched with JDK1.8 as a classic Spring Web project with WAR deployment, we now have a Spring Boot application with integrated Tomcat 9 on JDK11.

I use AdaptOpenJdk V11 here because I need Longtermsupport so I do not have to rebuild my blog every 6 months.

morphium

Of course, this should also be a small "showcase" for morphium, so here are a few examples:

Translations

In order to keep the application simple multilingual, it is advisable to keep the translation data in the database. This has many advantages:

  • You can change the translations at runtime
  • You can also adjust number formats, etc. at runtime and thus change the appearance of the application
  • if you allow it, you can also incorporate links, IMG tags etc in the translation table and thus achieve even more flexibility

Of course these features come with a price tag:

  • a little programming effort
  • The performance could suffer because for each page build, multiple data must be read from the database. This is where the caching feature of morphium helps us
  • You always need a database to get a page to face, otherwise it comes out nonsense. And the care can be a bit awkward.

Just the last point annoyed me, so I wrote Localization so that in the case that no occupation is found, the link to the appropriate maintenance interface is issued. Although this had its strange effects (link in a link, link in the title etc), was on the whole but really easy.

An important point was that I can only show the entries in the translation mask with a mouse click, which still lacks a translation. This is a quick way to a complete translation.

This is the message source:

@Service
public class LocalizationMessageSource extends AbstractMessageSource {
    @Autowired
    private LocalizationService localizationService;


    @Override
    protected MessageFormat resolveCode(String s, Locale locale) {

        String lang = locale.getLanguage();
        if (!LocalizationService.supportedLangs.contains(lang)) {
            return null;
        }
        return createMessageFormat(localizationService.getText(lang, s), locale);
    }


}

And this is the service returning the html-link if the translation is missing:


@Service
public class LocalizationService {
    public final static List<String> supportedLangs = Arrays.asList("de", "en");

    @Autowired
    private Morphium morphium;


    public String getText(String locale, String key) {
        key = key.replaceAll("[ \"'?!`´<>(){}-]", "_");
        Localization lz = morphium.findById(Localization.class, key);
        if (lz == null) {
            lz = new Localization();
            lz.setKey(key);
            for (String l : supportedLangs) {

                lz.getTxt().put(l, l + "_" + key);

            }
            morphium.store(lz);
        }

        if (key.startsWith("format_") && lz.getTxt().get(locale).equals(locale + "_" + key)) {
            lz.getTxt().put(locale, "yyyy-MM-dd HH:mm:ss 'default'");
            morphium.store(lz);
            return lz.getTxt().get(locale);
        }

        if (lz.getTxt().get(locale).equals(locale+"_"+key)){
            return "<a href=/admin/translations/query?queryKey="+key+">"+key+"</a>";
        }
        return lz.getTxt().get(locale);
    }


}

This is very handy if you bring in new features.

If you did that for a bigger website, it could be awkward. Especially if you want to support more than 2 languages. Therefore, in such a case, an import / export of the data is recommended, preferably in a readable format for translation agencies!

There is another "problem" in translations: orphans! It happens quite often that during the development, translations are made for something that becomes obsolete over time. Then you have translations in the database that nobody needs anymore. It is not so easy to identify them because the keys for the translations are partly dependent on data and only available at runtime.

For such a case, morphine offers the option of placing a lastAccessTimestamp in a field. This will let you see when entries were last read. (Attention: use this only in conjunction with caching, since every read access to these elements has a write access result!).

If you put it all together, the following entity comes out:

@Entity
@Cache(maxEntries = 10000, timeout = 60 * 60 * 1000, clearOnWrite = true, strategy = Cache.ClearStrategy.FIFO, syncCache = Cache.SyncCacheStrategy.CLEAR_TYPE_CACHE)
@LastAccess
public class Localization {
    @Id
    private String key;
    @LastAccess
    @Index
    private long lastAccess;
    private Map<String, String> txt;


    public String getKey() {
        return key;
    }

    public void setKey(String key) {
        this.key = key;
    }

    public Map<String, String> getTxt() {
        if (txt == null) txt = new HashMap<>();
        return txt;
    }

    public void setTxt(Map<String, String> txt) {
        this.txt = txt;
    }

    public long getLastAccess() {
        return lastAccess;
    }

    public void setLastAccess(long lastAccess) {
        this.lastAccess = lastAccess;
    }
}

If there is a write access to the translations, the cache is completely emptied.

If you use Jblog in the cluster, you need a way to keep the caches in sync on the nodes. For this you define a SyncCacheStrategy. The following happens:

  • all Jblog instances initialize a messaging system in morphine
  • every update on cached elements sends an update message via the messaging
  • depending on the strategy, the individual nodes update their cache.
  • SyncCacheStrategy.CLEAR_TYPE_CACHE clears the whole cache for this entity
  • SyncCacheStrategy.REMOVE_ENTRY_FROM_TYPE_CACHE removes the entry from all search results in the cache (Caution: the element will not be output in cached search results, although it might fit)
  • SyncCacheStrategy.UPDATE_ENTRY updates the entry in the cache (Caution: can cause the item in the cache to be counted as search results, even though it does not really fit the criteria anymore - dirty read!)
  • SyncCacheStrategy.NONE - well ... just do not sync: smirk:

But for this to work properly, the application needs to know which language is being used. Various steps are used for this:

  • is in the URL a lang parameter in it, overwrites everything that is set
  • is there a cookie with the language, then we take that
  • If the browser sends info about the supported languages, we use that.
  • if everything fails, take English: smirk:

This was implemented with an interceptor:


@Component
public class ValidLocaleCheckInterceptor extends HandlerInterceptorAdapter {
    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
        String lang = "de";
        if (request.getParameter("lang") != null) {
            String ln = request.getParameter("lang");
            if (ln.contains("_")) {
                ln = ln.substring(0, ln.indexOf("_"));
            }
            ln = ln.toLowerCase();
            if (ln.isEmpty()) {
                ln = "de";
            }

            if (!LocalizationService.supportedLangs.contains(ln)) {
                StringBuilder url = new StringBuilder();
                url.append(request.getRequestURI());
                Map<String, String[]> map = request.getParameterMap();
                String sep = "?";
                for (String n : map.keySet()) {
                    if (n.equals("lang")) {
                        continue;
                    }
                    for (String v : map.get(n)) {
                        url.append(sep);
                        sep = "&";
                        url.append(n);
                        url.append("=");
                        url.append(v);
                    }

                }
                url.append(sep);
                url.append("lang=de")//Default locale
                response.sendRedirect(url.toString());
                //Utils.showError(404,"unsupported language",response);
                return false;
            }
            lang = ln;
        } else {
            Cookie[] cookies = request.getCookies();
            boolean found = false;
            if (cookies != null) {
                for (int i = 0; i < cookies.length; i++) {
                    if (cookies[i].getName().equals("lang")) {
                        lang = cookies[i].getValue();
                        found = true;
                        break;
                    }
                }
            }
            if (!found) {
                LocaleResolver localeResolver = RequestContextUtils.getLocaleResolver(request);
                Locale l = localeResolver.resolveLocale(request);
                if (l != null) {
                    lang = l.getLanguage();
                }
            }
        }
        request.getSession().setAttribute("lang", lang);
        Cookie c = new Cookie("lang", lang);
        c.setPath("/");
        c.setMaxAge(365 * 24 * 60 * 60);
        response.addCookie(c);
        return true;
    }

Tag Cloud

To calculate the tag cloud we simply use the aggregator located in Mongo, and of course there is also support for that in morphium:

 Aggregator<BlogEntry,WordCloud> a=morphium.createAggregator(BlogEntry.class,WordCloud.class);
 a.match(morphium.createQueryFor(BlogEntry.class).f(BlogEntry.Fields.state).eq(Status.LIVE).f("wl").eq(wl));

 Map<String,Object> projection=new HashMap<>();
 Map<String,Object> arrayElemAt=new HashMap<>();
 ArrayList<Object> l = new ArrayList<>();
 l.add("$category_path");
 l.add(0);
 arrayElemAt.put("$arrayElemAt", l);
 projection.put("category",arrayElemAt);
 a.project(projection);

 a.group("$category").sum("count",1).end();
 a.sort("-count");
 log.debug("Preparing category cloud");
 List<WordCloud> cCloud = a.aggregate();

Here a simple aggregation is made on the BlogEntry entries stored in Mongo. The whole thing is filtered to white label (wl) (Jblog supports several whitelabel, for me it is boesebeck.biz, boesebeck.name and caluga.de) and only posts in the statusLIVE are considered.

On this data, a projection is performed to filter out the individual categories ($ arrayElementAt). Then we still count the occurrences of these categories and voila! A cloud ...

and so that the whole is mapped correctly in Java, you can put the result of the aggregation into an @Entity:

    @Entity
    public class WordCloud {
        @Id
        public String word;
        public int count;
        public int sz=0;

        public String getWord() {
            return word;
        }

        public void setWord(String word) {
            this.word = word;
        }

        public int getCount() {
            return count;
        }

        public void setCount(int count) {
            this.count = count;
        }

        public int getSz() {
            return sz;
        }

        public void setSz(int sz) {
            this.sz = sz;
        }
    }

Now only the list of WordCounts is displayed in the frontend, with the corresponding size Sz - and to make it look "funny", the whole thing is still randomized ...

Finished!

caching

Caching uses Jblog quite often to keep performance high. In particular, the translations are cached. This gives you the ability to change the translations at any time via a frontend, without having to re-deploy or restart, on the other hand, these data are rarely changed, so it is good to keep them in memory to access the database minimize and thus maximize performance.

In Morphium this is quite simple: just add the annotation @Cache to any Entity.

@Cached
@Entity
public class Translation {
...
}

statistics

Apart from the fact that morphium has its own statistics e.g. the cache efficiency of individual entities (to be retrieved via morphium.getStatistics (); - returns a Map <String, Double> with particular number of elements in the cache, cache hit ratio, etc.), so needs a blog but a few Information about the accesses to it.

The whole is handled in Jblog on its own service, which counts the daily accesses:

  • The stats are stored in a separate collection, and the date is used as _id
  • on the stats are then run together with the date inc commandos to increase the number of hits / visitors:` morphium.createQueryFor (Stats.class) .f (Stats.Fields.id) .eq (dateString) .inc (Stats.Fields.hits, 1); '
  • so that the accesses are not falsified by bots, you should filter out bots largely. To do this, look at the request header, the UserAgent provides information (at least a little - you do not get caught 100%).
  • I also log the maximum number Conccurent User. According to my definition for my applications, this is the number of unique session ids that have had a hit in the last 3 minutes. To measure this, the application must carry timestamps and aggregate the data accordingly.

To pack that into the mongo is a bit awkward and is enough for my mini-blog. If I had more traffic, I would not do it that way, but use a dedicated Time Series DB, like Influx or graphite.

Versioning of BlogPosts

This is a feature that does not really need it and can certainly be ticked off under "over-engineering". The idea: you have a history of changes for each block post and can jump back to every change. That's quite nice, but in my case completely useless: smirk:

Here is the entity for a BlogPost:

@Entity
@CreationTime
@LastChange
@Lifecycle
@Cache(timeout = 60000 * 10, maxEntries = 1000)
@Index(value = {"text.de:text,text.en:text,title.de:text,title.en:text", "state,visible_since"})
public class BlogEntry extends BlogEntryEmbedded {
    @Id
    protected MorphiumId id;
    private List<BlogEntryEmbedded> revisions;
    private BlogEntryEmbedded currentEdit;

    public List<BlogEntryEmbedded> getRevisions() {
        return revisions;
    }

    public void setRevisions(List<BlogEntryEmbedded> revisions) {
        this.revisions = revisions;
    }

    public BlogEntryEmbedded getCurrentEdit() {
        return currentEdit;
    }

    public void setCurrentEdit(BlogEntryEmbedded currentEdit) {
        this.currentEdit = currentEdit;
    }

    public MorphiumId getId() {
        return id;
    }

    public void setId(MorphiumId id) {
        this.id = id;
    }

    @PreStore
    public void preStore() {
        if (year == 0) {
            GregorianCalendar cal = new GregorianCalendar();
            year = cal.get(Calendar.YEAR);
            month = cal.get(Calendar.MONTH) + 1;
            day = cal.get(Calendar.DAY_OF_MONTH);
        }
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) {
            return true;
        }
        //        if (!(o instanceof BlogEntry)) {
        //            return false;
        //        }
        if (!super.equals(o)) {
            return false;
        }

        BlogEntry entry (BlogEntry) o;

        if (id != null ? !id.equals(entry.id) : entry.id != null) {
            return false;
        }
        if (revisions != null ? !revisions.equals(entry.revisions) : entry.revisions != null) {
            return false;
        }
        return currentEdit != null ? currentEdit.equals(entry.currentEdit) : entry.currentEdit == null;
    }

    @Override
    public int hashCode() {
        int result = super.hashCode();
        result = 31 * result + (id != null ? id.hashCode() : 0);
        result = 31 * result + (revisions != null ? revisions.hashCode() : 0);
        result = 31 * result + (currentEdit != null ? currentEdit.hashCode() : 0);
        return result;
    }

    public enum Fields {tweetAboutIt, tweetedAt, titleEn, creator, created, lastUpdate, year, month, day, tags, textDe, textEn, renderedPreviewDe, renderedPreviewEn, visibleSince, state, categoryPath, titleEnEscaped, titleDeEscaped, titleDe, revisions, tweetId, id}
}

and the embedded revisions:

 @Embedded
public class BlogEntryEmbedded {

    protected Map<String, String> title;
    //@Reference(lazyLoading = true)
    protected MorphiumId creator;

    @Index
    protected String wl;

    @CreationTime
    protected Long created;
    @LastChange
    protected Long lastUpdate;
    protected int year;
    protected int month;
    protected int day;
    protected List<String> tags;


    protected Map<String, String> text;
    protected Map<String, String> renderedPreview;
    protected long visibleSince;
    @Index
    protected Status state = Status.NEW;
    private List<String> categoryPath;
    @Index
    private Map<String, String> titleEscaped;
    private boolean tweetAboutIt;
    private long tweetedAt;
    private long tweetId;
    private List<String> additionalVisibleOn;

    public MorphiumId getCreator() {
        return creator;
    }

    public void setCreator(MorphiumId creator) {
        this.creator = creator;
    }

    public long getTweetId() {
        return tweetId;
    }

    public void setTweetId(long tweetId) {
        this.tweetId = tweetId;
    }

    public Long getCreated() {
        return created;
    }

    public void setCreated(Long created) {
        this.created = created;
    }

    public Long getLastUpdate() {
        return lastUpdate;
    }

    public void setLastUpdate(Long lastUpdate) {
        this.lastUpdate = lastUpdate;
    }

    public int getYear() {
        return year;
    }

    public void setYear(int year) {
        this.year = year;
    }

    public int getMonth() {
        return month;
    }

    public void setMonth(int month) {
        this.month = month;
    }

    public int getDay() {
        return day;
    }

    public void setDay(int day) {
        this.day = day;
    }

    public List<String> getTags() {
        if (tags == null) tags = new ArrayList<>();
        return tags;
    }

    public void setTags(List<String> tags) {
        this.tags = tags;
    }



    public long getVisibleSince() {
        return visibleSince;
    }

    public void setVisibleSince(long visibleSince) {
        this.visibleSince = visibleSince;
    }

    public Status getState() {
        return state;
    }

    public void setState(Status state) {
        this.state = state;
    }

    public Map<String, String> getTitle() {
        if (title == null) title = new HashMap<>();
        return title;
    }

    public void setTitle(Map<String, String> title) {
        this.title = title;
    }

    public Map<String, String> getText() {
        if (text == null) text = new HashMap<>();
        return text;
    }

    public void setText(Map<String, String> text) {
        this.text = text;
    }

    public Map<String, String> getRenderedPreview() {
        if (renderedPreview == null) renderedPreview = new HashMap<>();
        return renderedPreview;
    }

    public void setRenderedPreview(Map<String, String> renderedPreview) {
        this.renderedPreview = renderedPreview;
    }

    public List<String> getCategoryPath() {
        if (categoryPath == null) categoryPath = new ArrayList<>();
        return categoryPath;
    }

    public void setCategoryPath(List<String> categoryPath) {
        this.categoryPath = categoryPath;
    }

    public Map<String, String> getTitleEscaped() {
        if (titleEscaped == null) titleEscaped = new HashMap<>();
        return titleEscaped;
    }

    public void setTitleEscaped(Map<String, String> titleEscaped) {
        this.titleEscaped = titleEscaped;
    }

    public String getWl() {
        return wl;
    }

    public void setWl(String wl) {
        this.wl = wl;
    }

    public boolean isTweetAboutIt() {
        return tweetAboutIt;
    }

    public void setTweetAboutIt(boolean tweetAboutIt) {
        this.tweetAboutIt = tweetAboutIt;
    }

    public long getTweetedAt() {
        return tweetedAt;
    }

    public void setTweetedAt(long tweetedAt) {
        this.tweetedAt = tweetedAt;
    }
    
    public List<String> getAdditionalVisibleOn() {
        if (additionalVisibleOn == null) additionalVisibleOn = new ArrayList<>();
        return additionalVisibleOn;
    }

    public void setAdditionalVisibleOn(List<String> additionalVisibleOn) {
        this.additionalVisibleOn = additionalVisibleOn;
    }
}

As mentioned in the introduction, this is a feature that I will probably expand again. It does not really bring much. But it's nice to have something like that implemented.

Conclusion

Jblog is certainly not suitable for every blogger, especially the many features, the adaptability of e.g. Wordpress are completely missing here! If I want to make my blog look different, I have to put myself ran and rebuild the thing. There are no plugins.

However, I have to say that I've had virtually no problems with Jblog since it's been used. And that's also something ... for me as a private hobby blogger certainly one of the most important, if not the most important feature at all.

If anyone is interested in taking a closer look at jblog, I'll put it online again - just leave a comment here, or send me a mail. At the moment the whole project is running as a private project in GitHub, because it is so simple, e.g. to work with deployment keys etc.

created Stephan Bösebeck (stephan)