latest blog entries

latest updates

Category Cloud:
MongoDB Computer Linux/Unix Tweet global Java Apple

Tag Cloud:
markdown morphium java-2 ergodox-ez blog git objective-c-2 morphium-2 security java mongo objective-c rsa jblog english ios encryption cache ergodox qnap POJO mongodb storage memorymanagement tweet programming

category: Computer --> programming --> MongoDB --> morphium

Morphium messaging JMS implementation

2019-08-20 - Tags: morphium java

Morphium 4.0.6.2 was released this week and this version contains one feature, which I want to show here: Morphium Messaging via JMS!

Here a little proof of concept test from the source:

public class BasicJMSTests extends MongoTest {

    @Test
    public void basicSendReceiveTest() throws Exception {
        JMSConnectionFactory factory = new JMSConnectionFactory(morphium);
        JMSContext ctx1 = factory.createContext();
        JMSContext ctx2 = factory.createContext();

        JMSProducer pr1 = ctx1.createProducer();
        Topic dest = new JMSTopic("test1");

        JMSConsumer con = ctx2.createConsumer(dest);
        con.setMessageListener(message -> log.info("Got Message!"));
        pr1.send(dest, "A test");

        ctx1.close();
        ctx2.close();
    }

    @Test
    public void synchronousSendRecieveTest() throws Exception {
        JMSConnectionFactory factory = new JMSConnectionFactory(morphium);
        JMSContext ctx1 = factory.createContext();
        JMSContext ctx2 = factory.createContext();

        JMSProducer pr1 = ctx1.createProducer();
        Topic dest = new JMSTopic("test1");
        JMSConsumer con = ctx2.createConsumer(dest);

        final Map<String, Object> exchange = new ConcurrentHashMap<>();
        Thread senderThread = new Thread(() -> {
            JMSTextMessage message = new JMSTextMessage();
            try {
                message.setText("Test");
            } catch (JMSException e) {
                e.printStackTrace();
            }
            pr1.send(dest, message);
            log.info("Sent out message");
            exchange.put("sent", true);
        });
        Thread receiverThread = new Thread(() -> {
            log.info("Receiving...");
            Message msg = con.receive();
            log.info("Got incoming message");
            exchange.put("received", true);
        });
        receiverThread.start();
        senderThread.start();
        Thread.sleep(5000);
        assert (exchange.get("sent") != null);
        assert (exchange.get("received") != null);
    }
}

Consider this only being a teaser, it is still BETA or even ALPHA. Not production ready yet.

Unfortunately, some features of morphium won't work using JMS:

via JMS you probably can't send direct messages to nodes
via JMS you can't youse custom message classes
in total, JMS is a lot less flexible than morphium messaging itself it seems. Morphium Messaging allows a lot more details in definition, what message is sent where and how.

But again: this is not production ready, BETA state at best!

category: Computer --> programming --> MongoDB --> morphium

Morphium based blogsoftware

2019-07-29 - Tags: jblog

Morphium based blog software Jblog

Some time ago I "dared" to leap away from Wordpress and reported about it (https://caluga.de/v/2017/5/19/jblog_java_blogging_software). Now it's been over 2 years (: astonished :) and there it is time to draw a conclusion and to report a little how it went.

Jblog - the good the bad the ugly

The good anticipation: the server was not hacked. The hits that were aimed at any PHP gaps or so are all ineffective "bounced" - that's what I wanted to achieve. The update topic is also solved: there was no: smirk: I took care of the further development myself. However, we already had a few bugs, and some are a bit annoying. Also, some things have turned out to be less practical. The quite elaborate versioning of the individual blog posts is actually unnecessary and complicates everything only.

But otherwise...

Technology

What is currently behind Jblog:

Morphium V4.0.6
Spring boat
Apache Freemarker
Bootstrap 4
JDK11
MongoDB 4.0

yes, that was it already, has become quite clear. That The application is currently running as a Spring Boot app behind a nginx.

features

I knit my blog as ich wanted it. So very much in my way of blogging, which may be the reason why nobody forked on Github and I changed the project to privat. : Sob:

multilingualism

Jblog is consistently bilingual. Theoretically, more would be possible, but I wanted to focus on German and English. That every element on the website is translated, all blog entries must be recorded in 2 languages.

For this a MessageSource is used which reads the data from Mongo. The nice: if a translation is not found, a link to the translation mask will be issued. That when I run Jblog on an empty translation database, I get a screen full of links that take me to the translation tool. Gradually this will be corrected.

By using Morphium to access the Mongo, these translations can also be cached (just write the @Cache annotation to the class).

The nice thing is that the cache, if wanted, is cluster-aware. That I can run multiple instances of Jblog and access through a loadbalancer. And these would keep their caches in sync across the Mongo, i. there are no dirty reads even with round-robin attitude!

Markdown

Markdown is indeed der hit lately to enter text. I think that's pretty cool too, especially since I type very well and the grip on the mouse always slows me down. That's why it's so easy to type everything. (and who like me used to do word processing with LaTeX knows this: smirk :.)

Jblog is completely designed for Markdown. That You enter his blogs in Markdown and then see the preview right away. I have also made some extensions, especially to simply emojis (: confused:: smile:: sweat :) integrate or embed a nicely formatted Java code block.

I also implemented the embedding of pictures.

I used a Markdown implementation called [Markingbird] (https://github.com/pmattos/Markingbird) and adjusted it a bit. So I've built something to embed the emojis easier and easier to view images.

Each blog post is written in Markdown and will be rendered on first access. And the result is cached, of course: smirk:

When typing, I have a live preview and can see exactly how the post will look at the end.

Twitter connection

new articles are published via Twitter. Some things have been taken into consideration: specify the time of the tweet, only for new articles etc.

And it is tweeted in 2 languages, German and English

multiuser capability

Yes, you can actually use Jblog with several users. Although I have a few friends and acquaintances, the time here from time to time pure snow and possibly what to post, but otherwise ... Is so a very little used feature, which is why I have then deactivated.

At the moment you can not register, the registration process is De-Enabled. If you want to leave a comment, you can do that via Disqus.

Implementation

The implementation has changed a bit in the last 2 years. Launched with JDK1.8 as a classic Spring Web project with WAR deployment, we now have a Spring Boot application with integrated Tomcat 9 on JDK11.

I use AdaptOpenJdk V11 here because I need Longtermsupport so I do not have to rebuild my blog every 6 months.

morphium

Of course, this should also be a small "showcase" for morphium, so here are a few examples:

Translations

In order to keep the application simple multilingual, it is advisable to keep the translation data in the database. This has many advantages:

You can change the translations at runtime
You can also adjust number formats, etc. at runtime and thus change the appearance of the application
if you allow it, you can also incorporate links, IMG tags etc in the translation table and thus achieve even more flexibility

Of course these features come with a price tag:

a little programming effort
The performance could suffer because for each page build, multiple data must be read from the database. This is where the caching feature of morphium helps us
You always need a database to get a page to face, otherwise it comes out nonsense. And the care can be a bit awkward.

Just the last point annoyed me, so I wrote Localization so that in the case that no occupation is found, the link to the appropriate maintenance interface is issued. Although this had its strange effects (link in a link, link in the title etc), was on the whole but really easy.

An important point was that I can only show the entries in the translation mask with a mouse click, which still lacks a translation. This is a quick way to a complete translation.

This is the message source:

@Service
public class LocalizationMessageSource extends AbstractMessageSource {
    @Autowired
    private LocalizationService localizationService;

    @Override
    protected MessageFormat resolveCode(String s, Locale locale) {

        String lang = locale.getLanguage();
        if (!LocalizationService.supportedLangs.contains(lang)) {
            return null;
        }
        return createMessageFormat(localizationService.getText(lang, s), locale);
    }

}

And this is the service returning the html-link if the translation is missing:

@Service
public class LocalizationService {
    public final static List<String> supportedLangs = Arrays.asList("de", "en");

    @Autowired
    private Morphium morphium;

    public String getText(String locale, String key) {
        key = key.replaceAll("[ \"'?!`´<>(){}-]", "_");
        Localization lz = morphium.findById(Localization.class, key);
        if (lz == null) {
            lz = new Localization();
            lz.setKey(key);
            for (String l : supportedLangs) {

                lz.getTxt().put(l, l + "_" + key);

            }
            morphium.store(lz);
        }

        if (key.startsWith("format_") && lz.getTxt().get(locale).equals(locale + "_" + key)) {
            lz.getTxt().put(locale, "yyyy-MM-dd HH:mm:ss 'default'");
            morphium.store(lz);
            return lz.getTxt().get(locale);
        }

        if (lz.getTxt().get(locale).equals(locale+"_"+key)){
            return "<a href=/admin/translations/query?queryKey="+key+">"+key+"</a>";
        }
        return lz.getTxt().get(locale);
    }

}

This is very handy if you bring in new features.

If you did that for a bigger website, it could be awkward. Especially if you want to support more than 2 languages. Therefore, in such a case, an import / export of the data is recommended, preferably in a readable format for translation agencies!

There is another "problem" in translations: orphans! It happens quite often that during the development, translations are made for something that becomes obsolete over time. Then you have translations in the database that nobody needs anymore. It is not so easy to identify them because the keys for the translations are partly dependent on data and only available at runtime.

For such a case, morphine offers the option of placing a lastAccessTimestamp in a field. This will let you see when entries were last read. (Attention: use this only in conjunction with caching, since every read access to these elements has a write access result!).

If you put it all together, the following entity comes out:

@Entity
@Cache(maxEntries = 10000, timeout = 60 * 60 * 1000, clearOnWrite = true, strategy = Cache.ClearStrategy.FIFO, syncCache = Cache.SyncCacheStrategy.CLEAR_TYPE_CACHE)
@LastAccess
public class Localization {
    @Id
    private String key;
    @LastAccess
    @Index
    private long lastAccess;
    private Map<String, String> txt;

    public String getKey() {
        return key;
    }

    public void setKey(String key) {
        this.key = key;
    }

    public Map<String, String> getTxt() {
        if (txt == null) txt = new HashMap<>();
        return txt;
    }

    public void setTxt(Map<String, String> txt) {
        this.txt = txt;
    }

    public long getLastAccess() {
        return lastAccess;
    }

    public void setLastAccess(long lastAccess) {
        this.lastAccess = lastAccess;
    }
}

If there is a write access to the translations, the cache is completely emptied.

If you use Jblog in the cluster, you need a way to keep the caches in sync on the nodes. For this you define a SyncCacheStrategy. The following happens:

all Jblog instances initialize a messaging system in morphine
every update on cached elements sends an update message via the messaging
depending on the strategy, the individual nodes update their cache.
SyncCacheStrategy.CLEAR_TYPE_CACHE clears the whole cache for this entity
SyncCacheStrategy.REMOVE_ENTRY_FROM_TYPE_CACHE removes the entry from all search results in the cache (Caution: the element will not be output in cached search results, although it might fit)
SyncCacheStrategy.UPDATE_ENTRY updates the entry in the cache (Caution: can cause the item in the cache to be counted as search results, even though it does not really fit the criteria anymore - dirty read!)
SyncCacheStrategy.NONE - well ... just do not sync: smirk:

But for this to work properly, the application needs to know which language is being used. Various steps are used for this:

is in the URL a lang parameter in it, overwrites everything that is set
is there a cookie with the language, then we take that
If the browser sends info about the supported languages, we use that.
if everything fails, take English: smirk:

This was implemented with an interceptor:

@Component
public class ValidLocaleCheckInterceptor extends HandlerInterceptorAdapter {
    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
        String lang = "de";
        if (request.getParameter("lang") != null) {
            String ln = request.getParameter("lang");
            if (ln.contains("_")) {
                ln = ln.substring(0, ln.indexOf("_"));
            }
            ln = ln.toLowerCase();
            if (ln.isEmpty()) {
                ln = "de";
            }

            if (!LocalizationService.supportedLangs.contains(ln)) {
                StringBuilder url = new StringBuilder();
                url.append(request.getRequestURI());
                Map<String, String[]> map = request.getParameterMap();
                String sep = "?";
                for (String n : map.keySet()) {
                    if (n.equals("lang")) {
                        continue;
                    }
                    for (String v : map.get(n)) {
                        url.append(sep);
                        sep = "&";
                        url.append(n);
                        url.append("=");
                        url.append(v);
                    }

                }
                url.append(sep);
                url.append("lang=de"); //Default locale
                response.sendRedirect(url.toString());
                //Utils.showError(404,"unsupported language",response);
                return false;
            }
            lang = ln;
        } else {
            Cookie[] cookies = request.getCookies();
            boolean found = false;
            if (cookies != null) {
                for (int i = 0; i < cookies.length; i++) {
                    if (cookies[i].getName().equals("lang")) {
                        lang = cookies[i].getValue();
                        found = true;
                        break;
                    }
                }
            }
            if (!found) {
                LocaleResolver localeResolver = RequestContextUtils.getLocaleResolver(request);
                Locale l = localeResolver.resolveLocale(request);
                if (l != null) {
                    lang = l.getLanguage();
                }
            }
        }
        request.getSession().setAttribute("lang", lang);
        Cookie c = new Cookie("lang", lang);
        c.setPath("/");
        c.setMaxAge(365 * 24 * 60 * 60);
        response.addCookie(c);
        return true;
    }

To calculate the tag cloud we simply use the aggregator located in Mongo, and of course there is also support for that in morphium:

Aggregator<BlogEntry,WordCloud> a=morphium.createAggregator(BlogEntry.class,WordCloud.class);
a.match(morphium.createQueryFor(BlogEntry.class).f(BlogEntry.Fields.state).eq(Status.LIVE).f("wl").eq(wl));

Map<String,Object> projection=new HashMap<>();
Map<String,Object> arrayElemAt=new HashMap<>();
ArrayList<Object> l = new ArrayList<>();
l.add("$category_path");
l.add(0);
arrayElemAt.put("$arrayElemAt", l);
projection.put("category",arrayElemAt);
a.project(projection);

a.group("$category").sum("count",1).end();
a.sort("-count");
log.debug("Preparing category cloud");
List<WordCloud> cCloud = a.aggregate();

Here a simple aggregation is made on the BlogEntry entries stored in Mongo. The whole thing is filtered to white label (wl) (Jblog supports several whitelabel, for me it is boesebeck.biz, boesebeck.name and caluga.de) and only posts in the statusLIVE are considered.

On this data, a projection is performed to filter out the individual categories ($ arrayElementAt). Then we still count the occurrences of these categories and voila! A cloud ...

and so that the whole is mapped correctly in Java, you can put the result of the aggregation into an @Entity:

    @Entity
    public class WordCloud {
        @Id
        public String word;
        public int count;
        public int sz=0;

        public String getWord() {
            return word;
        }

        public void setWord(String word) {
            this.word = word;
        }

        public int getCount() {
            return count;
        }

        public void setCount(int count) {
            this.count = count;
        }

        public int getSz() {
            return sz;
        }

        public void setSz(int sz) {
            this.sz = sz;
        }
    }

Now only the list of WordCounts is displayed in the frontend, with the corresponding size Sz - and to make it look "funny", the whole thing is still randomized ...

Finished!

caching

Caching uses Jblog quite often to keep performance high. In particular, the translations are cached. This gives you the ability to change the translations at any time via a frontend, without having to re-deploy or restart, on the other hand, these data are rarely changed, so it is good to keep them in memory to access the database minimize and thus maximize performance.

In Morphium this is quite simple: just add the annotation @Cache to any Entity.

@Cached
@Entity
public class Translation {
...
}

statistics

Apart from the fact that morphium has its own statistics e.g. the cache efficiency of individual entities (to be retrieved via morphium.getStatistics (); - returns a Map <String, Double> with particular number of elements in the cache, cache hit ratio, etc.), so needs a blog but a few Information about the accesses to it.

The whole is handled in Jblog on its own service, which counts the daily accesses:

The stats are stored in a separate collection, and the date is used as _id
on the stats are then run together with the date inc commandos to increase the number of hits / visitors:` morphium.createQueryFor (Stats.class) .f (Stats.Fields.id) .eq (dateString) .inc (Stats.Fields.hits, 1); '
so that the accesses are not falsified by bots, you should filter out bots largely. To do this, look at the request header, the UserAgent provides information (at least a little - you do not get caught 100%).
I also log the maximum number Conccurent User. According to my definition for my applications, this is the number of unique session ids that have had a hit in the last 3 minutes. To measure this, the application must carry timestamps and aggregate the data accordingly.

To pack that into the mongo is a bit awkward and is enough for my mini-blog. If I had more traffic, I would not do it that way, but use a dedicated Time Series DB, like Influx or graphite.

Versioning of BlogPosts

This is a feature that does not really need it and can certainly be ticked off under "over-engineering". The idea: you have a history of changes for each block post and can jump back to every change. That's quite nice, but in my case completely useless: smirk:

Here is the entity for a BlogPost:

@Entity
@CreationTime
@LastChange
@Lifecycle
@Cache(timeout = 60000 * 10, maxEntries = 1000)
@Index(value = {"text.de:text,text.en:text,title.de:text,title.en:text", "state,visible_since"})
public class BlogEntry extends BlogEntryEmbedded {
    @Id
    protected MorphiumId id;
    private List<BlogEntryEmbedded> revisions;
    private BlogEntryEmbedded currentEdit;

    public List<BlogEntryEmbedded> getRevisions() {
        return revisions;
    }

    public void setRevisions(List<BlogEntryEmbedded> revisions) {
        this.revisions = revisions;
    }

    public BlogEntryEmbedded getCurrentEdit() {
        return currentEdit;
    }

    public void setCurrentEdit(BlogEntryEmbedded currentEdit) {
        this.currentEdit = currentEdit;
    }

    public MorphiumId getId() {
        return id;
    }

    public void setId(MorphiumId id) {
        this.id = id;
    }

    @PreStore
    public void preStore() {
        if (year == 0) {
            GregorianCalendar cal = new GregorianCalendar();
            year = cal.get(Calendar.YEAR);
            month = cal.get(Calendar.MONTH) + 1;
            day = cal.get(Calendar.DAY_OF_MONTH);
        }
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) {
            return true;
        }
        //        if (!(o instanceof BlogEntry)) {
        //            return false;
        //        }
        if (!super.equals(o)) {
            return false;
        }

        BlogEntry entry = (BlogEntry) o;

        if (id != null ? !id.equals(entry.id) : entry.id != null) {
            return false;
        }
        if (revisions != null ? !revisions.equals(entry.revisions) : entry.revisions != null) {
            return false;
        }
        return currentEdit != null ? currentEdit.equals(entry.currentEdit) : entry.currentEdit == null;
    }

    @Override
    public int hashCode() {
        int result = super.hashCode();
        result = 31 * result + (id != null ? id.hashCode() : 0);
        result = 31 * result + (revisions != null ? revisions.hashCode() : 0);
        result = 31 * result + (currentEdit != null ? currentEdit.hashCode() : 0);
        return result;
    }

    public enum Fields {tweetAboutIt, tweetedAt, titleEn, creator, created, lastUpdate, year, month, day, tags, textDe, textEn, renderedPreviewDe, renderedPreviewEn, visibleSince, state, categoryPath, titleEnEscaped, titleDeEscaped, titleDe, revisions, tweetId, id}
}

and the embedded revisions:

@Embedded
public class BlogEntryEmbedded {

    protected Map<String, String> title;
    //@Reference(lazyLoading = true)
    protected MorphiumId creator;

    @Index
    protected String wl;

    @CreationTime
    protected Long created;
    @LastChange
    protected Long lastUpdate;
    protected int year;
    protected int month;
    protected int day;
    protected List<String> tags;

    protected Map<String, String> text;
    protected Map<String, String> renderedPreview;
    protected long visibleSince;
    @Index
    protected Status state = Status.NEW;
    private List<String> categoryPath;
    @Index
    private Map<String, String> titleEscaped;
    private boolean tweetAboutIt;
    private long tweetedAt;
    private long tweetId;
    private List<String> additionalVisibleOn;

    public MorphiumId getCreator() {
        return creator;
    }

    public void setCreator(MorphiumId creator) {
        this.creator = creator;
    }

    public long getTweetId() {
        return tweetId;
    }

    public void setTweetId(long tweetId) {
        this.tweetId = tweetId;
    }

    public Long getCreated() {
        return created;
    }

    public void setCreated(Long created) {
        this.created = created;
    }

    public Long getLastUpdate() {
        return lastUpdate;
    }

    public void setLastUpdate(Long lastUpdate) {
        this.lastUpdate = lastUpdate;
    }

    public int getYear() {
        return year;
    }

    public void setYear(int year) {
        this.year = year;
    }

    public int getMonth() {
        return month;
    }

    public void setMonth(int month) {
        this.month = month;
    }

    public int getDay() {
        return day;
    }

    public void setDay(int day) {
        this.day = day;
    }

    public List<String> getTags() {
        if (tags == null) tags = new ArrayList<>();
        return tags;
    }

    public void setTags(List<String> tags) {
        this.tags = tags;
    }

    public long getVisibleSince() {
        return visibleSince;
    }

    public void setVisibleSince(long visibleSince) {
        this.visibleSince = visibleSince;
    }

    public Status getState() {
        return state;
    }

    public void setState(Status state) {
        this.state = state;
    }

    public Map<String, String> getTitle() {
        if (title == null) title = new HashMap<>();
        return title;
    }

    public void setTitle(Map<String, String> title) {
        this.title = title;
    }

    public Map<String, String> getText() {
        if (text == null) text = new HashMap<>();
        return text;
    }

    public void setText(Map<String, String> text) {
        this.text = text;
    }

    public Map<String, String> getRenderedPreview() {
        if (renderedPreview == null) renderedPreview = new HashMap<>();
        return renderedPreview;
    }

    public void setRenderedPreview(Map<String, String> renderedPreview) {
        this.renderedPreview = renderedPreview;
    }

    public List<String> getCategoryPath() {
        if (categoryPath == null) categoryPath = new ArrayList<>();
        return categoryPath;
    }

    public void setCategoryPath(List<String> categoryPath) {
        this.categoryPath = categoryPath;
    }

    public Map<String, String> getTitleEscaped() {
        if (titleEscaped == null) titleEscaped = new HashMap<>();
        return titleEscaped;
    }

    public void setTitleEscaped(Map<String, String> titleEscaped) {
        this.titleEscaped = titleEscaped;
    }

    public String getWl() {
        return wl;
    }

    public void setWl(String wl) {
        this.wl = wl;
    }

    public boolean isTweetAboutIt() {
        return tweetAboutIt;
    }

    public void setTweetAboutIt(boolean tweetAboutIt) {
        this.tweetAboutIt = tweetAboutIt;
    }

    public long getTweetedAt() {
        return tweetedAt;
    }

    public void setTweetedAt(long tweetedAt) {
        this.tweetedAt = tweetedAt;
    }

    public List<String> getAdditionalVisibleOn() {
        if (additionalVisibleOn == null) additionalVisibleOn = new ArrayList<>();
        return additionalVisibleOn;
    }

    public void setAdditionalVisibleOn(List<String> additionalVisibleOn) {
        this.additionalVisibleOn = additionalVisibleOn;
    }
}

As mentioned in the introduction, this is a feature that I will probably expand again. It does not really bring much. But it's nice to have something like that implemented.

Conclusion

Jblog is certainly not suitable for every blogger, especially the many features, the adaptability of e.g. Wordpress are completely missing here! If I want to make my blog look different, I have to put myself ran and rebuild the thing. There are no plugins.

However, I have to say that I've had virtually no problems with Jblog since it's been used. And that's also something ... for me as a private hobby blogger certainly one of the most important, if not the most important feature at all.

If anyone is interested in taking a closer look at jblog, I'll put it online again - just leave a comment here, or send me a mail. At the moment the whole project is running as a private project in GitHub, because it is so simple, e.g. to work with deployment keys etc.

category: Computer --> programming --> MongoDB --> morphium

Morphium 4.0.0 released

2018-11-09 - Tags: mongo mongodb java morphium

What was that again? Morphium is a sophisticated Object Mapper and Messaging System for MongoDB. It adds a lot of features to MongoDB, for example a dedicated InMemoryDriver, so that you can run all your Tests just in RAM without the need to install a MongoDB.

and good things need some time... so we are happy to announce, we just released Morphium 4.0.0. This release contains a ton of changes and improvements.

Simplifying the API at a lot of places in code
a lot new features and improvements for the Messaging System (Rejecting of Messages, Pausing/Unpausing of message processing, message replay upon startup, ...)
Code quality improvements
adding a new jackson based ObjectMapper (has to be enabled in Settings)
Transaction Support
Improvement of Enum handling
lots of improvements with aggregation
the InMemDriver does now support ChangeStreams
and countless other changes

this is a big update which took 8 Release Candidates to test.

Morphium is available at maven central:

<dependency>
    <groupId>de.caluga</groupId>
    <artifactId>morphium</artifactId>
    <version>4.0.0</version>
</dependency>

or at github.

category: Computer --> programming --> MongoDB --> morphium

Morphium V4.0.0-RC5

2018-10-19 - Tags: mongodb java morphium

We went quite a long way to get here, but... eventually we will

We put a lot of time and effort in this new Release Candidate #5 of Morphium and we get close to the major release.

what again was morphium?

Morphium was started as a simple Object mapper for Mongodb 1.0 which made querying in java simple using a fluent api. In addition there were a lot of unique features at that time like automatice dereferencing and lazy loading of references, caching, cluster aware cache synchronization etc.

Messaging was implemented in an early stage in order to get cache sync to work.

Messaging got more and more core feature of morphium and one USP of the API. So we gave this feature some love with this release...

Morphium V4.0.0

With this major release we added a lot of features and enhancements. We still need to adapt documentation though, but we will. And then we issue the release - promise

This is some of the new features of Morphium 4.0.0:

enhanced messaging. Using the watch functionality we can achieve a push functionality for messaging. Maximum performance, minimum load (only available in ReplicaSets)
added priorities in messages
re-implemented object mapper on base of jackson
stream-lined api, removed unused parts
full support for Mongodb 4.0 multi-document transactions

The codebase is already in a very good testet stage and is used in productional environments in some places.

Availability

Morphium is OpenSource and can be downloaded (including Releases and Release Candidates) from github.

Or, even easier, from maven central:

<dependency>
    <groupId>de.caluga</groupId>
    <artifactId>morphium</artifactId>
    <version>4.0.0-RC5</version>
</dependency>

category: Computer --> programming --> MongoDB --> morphium

Morphium 4.0.0 - work in progress

2018-08-15 - Tags: java mongodb morphium

We are still working on getting morphium 4.0.0 done. We are behind schedule a bit but want to explain here, what is going on at the moment:

complete new ObjectMapper implementation based on Jackson. Using this, we are about 30% faster on average (even twice as fast in some cases). And we use a standard lib here
new messaging functionalities. This is going to be the new core of Morphium, the Messaging! We put a lot of effort in this component, improved a lot, made it faster
de-cluttering of the API and of the code on a lot of places

Unfortunately V4.0.0 is not 100% compatible with the predecessors. It might be necessary to migrate data - although not very probably. If you need assistence with that, please contact us on google-groups or githup.

All of that is taking more time, than estimated. But BETA4 of morphium 4.0.0 can be downloaded on maven central:

 <dependency>
            <groupId>de.caluga</groupId>
            <artifactId>morphium</artifactId>
            <version>4.0.0-BETA4</version>
        </dependency>

In order for Morphium to work, you need to add the mongo drivers libs to your dependencies. We tested morphium with V3.8.0 of the drivers libs, but later versions should also work. (this is one of the reasons, why morphium does not create a direct dependency to the driver! Use the version you have in your project already, if not, use 3.8.0)

  <dependency>
            <groupId>org.mongodb</groupId>
            <artifactId>bson</artifactId>
            <version>3.8.0</version>
        </dependency>
        <dependency>
            <groupId>org.mongodb</groupId>
            <artifactId>mongodb-driver-core</artifactId>
            <version>3.8.0</version>
        </dependency>
        <dependency>
            <groupId>org.mongodb</groupId>
            <artifactId>mongodb-driver</artifactId>
            <version>3.8.0</version>
        </dependency>

Happy hacking!

category: global

Morphium 4.0.0 in the works

2018-07-17 - Tags: morphium java mongo

MongoDB released the long awaited V4.0 just recently. And of course there are a lot of new features, but the most exiting one was probably the support for multi document ACID transactions!

The mongo driver does support all those features since V3.8.0 - and morphium supports all of that with V4.0.0 - to match the mongodb version. The current BETA1 should be available via maven central or from github.com.

new Features in Morphium 4.0.0

Transaction Support

of course, this had to be added. But it is quite simple to use:

//start a transaction via
morphium.startTransaction();
//all calls in this thread use the transaction now

morphium.commitTransaction(); //commit
morphium.abortTransaction(); //abort

MorphiumTransactionContext ctx=morphium.getTransaction();

//to have other threads use the transaction, you need to pass on the transaction context
//Transactions are Threadlocal - so you need to set the context in threads accordingly

morphium.setTransaction(ctx);

Attention: if your mongodb does not support transactions, this will cause an exception

improved Messaging.

We put a lot of effort into messaging, simplified the API at some places and added some new features. Messaging now supports kind of "push" notifications, if connected to a replicaset.

Improved Cluster-Aware Caching

using the new watch functionality of mongod, which is nothing but another way to access the oplog, we could implement a feature to synchronize the cache via database push. It works similar as with the messaging, just that the messaging is not used. The notification about changes comes directly from database. the @Cache annotation is used to determine how to delete / update the cache.

we introduced a new cache synchronizer for that:

WatchingCacheSynchronizer sync = new WatchingCacheSynchronizer(morphium);
sync.start();

inMemoryDriver

We re-included and improved the In-Memory-Driver into morphium. So you can more easy Mock access to mongo, which is very important when testing.

MorphiumConfig cfg = new MorphiumConfig();
        cfg.addHostToSeed("inMem");
        cfg.setDatabase("test");
        cfg.setDriverClass(InMemoryDriver.class.getName());
        cfg.setReplicasetMonitoring(false);

Transactions are not working (yet) and aggregation might come in some later version of the driver. All other, more basic functionality is working.

category: global

Worauf es beim Homepage-Layout ankommt

2018-07-17 - Tags:

originally posted on: https://boesebeck.name

there is no english version of this text available

Anm.: Dieser Text wurde zur Verfügung gestellt von homepage-erstellen.de

Worauf es beim Homepage-Layout ankommt

Hat man erst einmal die ersten Hürden bei der Erstellung einer Homepage gemeistert, muss man sich um ein passendes Layout kümmern. Ein übersichtliches und ansprechendes Layout sorgt dafür, dass relevante Inhalte leichter gefunden werden können und Seitenbesucher eher zurückkehren.

Was ein gutes Layout auszeichnet

Beim Layout einer Homepage gilt es zunächst darauf zu achten, welchem Zweck die Homepage dienen soll. Soll ein Produkt vorgestellt werden? Möchte man über die Dienstleistung einer Firma informieren? Oder nutzt man die Homepage, um über ein persönliches Anliegen aufzuklären? Wichtig ist, dass alle relevanten Informationen jederzeit gefunden werden können. Ein gutes Layout besteht aus Überschriften, Bildern, Fußzeilen und Spalten. So werden Informationen sinnvoll vorgefiltert und können schon mit wenigen Blicken erfasst werden. Das erhöht für den Besucher der Seite den Bedienkomfort und die Wahrscheinlichkeit, dass man zu einem späteren Zeitpunkt die Seite nochmal aufsuchen wird. Zuerst werden beim Layout Farben und Formen wahrgenommen. Ein farbenfrohes Layout kann sich zum Beispiel für ein Portfolio eignen, das Kreativität ausdrücken soll, passt aber kaum zu bestimmten Firmen oder Dienstleistern. Bei diesen ist es wichtig, dass man die Informationen zu jedem Produkt sofort finden kann.

Seitenleiste zum Teil sehr nützlich

Laut www.homepage-erstellen.de kann sich eine Seitenleiste als sehr nützlich für Besucher der Seite erweisen. Dort sollten aber nicht die wichtigsten Inhalte, sondern hauptsächlich ergänzende Informationen zusammengefasst werden. Die Ausrichtung spielt dabei keine große Rolle und die Seitenleiste kann sowohl auf der rechten als auch auf der linken Seite angebracht werden. In der oberen linken Ecke sollte sich ein Logo befinden. Bei E-Commerce-Seiten ist der Warenkorb meist in der rechten Ecke angebracht. Das Suchfeld befindet sich oftmals direkt neben dem oder in direkter Nähe zum Warenkorb.

category: Computer --> programming --> MongoDB --> morphium

Custom Caching in Morphium

2018-05-20 - Tags: java mongodb morphium cache

since the first version of Morphium it provided an internal cache for all Entities maked with the annotation Cache. This cache was configured mainly via those annotations.

This cache has proven its usefulness in countless projects and the synchronizing of caches in clustered environments via Morphium Messaging is working flawlessly.

But there are new and more sophisticated Cache Implementations out there . It would not be clever to built all those features also into morphium, better leverage on those projects. So we decided to include JCache-Support (JSR107) into morphium.

Of course, we had to adapt some things here and there, especially the MorphiumCahce-Interface needed to be overhauled.

Morphium itself always had the option to use an own MorphiumCache Implementation. But this was not always easy to achieve in own projects. Hence we use that now in order to be able to offer both the old, proven implementation and the new, future-implementation.

As always, morphium can be used out of the box, so we implemented a JCAche-Version of our cache as well into morphium.

How to use

With the upcoming V3.2.2BETA1 (via maven central oder auf github ) morphium will use the JCache compatible implementation. If you want to switch back to the old, proved Version of caching, you just need to change the config:

MorphiumConfig cfg=new MorphiumConfig();
cfg.setCache(new MorphiumCacheImpl());

if you create your MorphiumConfig via properties or via JSon, you need to set the class name accordingly:

  cacheClassName=de.caluga.morphium.cache.MorphiumCacheImpl

JCache Support

If you leave all those settings to default, the JCache API is being used. By Default the cache creates the cache manager using Caching.getCachingProvider().getCacheManager(). This way you get the default of the default

If you want to configure the cache on your own (ehcache properties for example), you just need to pass on the CacheManager to the morphiumCache:

CachingProvider provider = Caching.getCachingProvider();
morphium.getCache.setCacheManager(provider.getCacheManager());

of course in this example, there are no additional options set, but I think you see, how that might work.

BTW: the morphium internal JCache implementation can be used via JCache API in your application also, if you want to. Just add the system setting -Djavax.cache.spi.CachingProvider=de.caluga.morphium.cache.jcache.CachingProviderImpl and with Caching.getCachingProvider() you will get the Morphium Implementation of the cache.

Attention All JCache implementation support expiration of oldest / least recently used entries in cache. Unfortunately the policy of morphium is a bit more complex (especially regarding the number of entries), so moprhium implements an own JCache-Housekeeping for now.

Additional Info: Whatever Cache Implementation you use, you might still use the CacheSynchronizer in order to synchronize caches. And this synchronization should be working via Mongo even if you are not storing any Entities using the cache as an application cache!

Maven settings

    <groupId>de.caluga</groupId>
    <artifactId>morphium</artifactId>
    <version>3.2.2BETA1</version>

known bugs

There are some minor known bugs in the current Beta, you might want to know:

the CacheListener Callbacks do not seem to work properly with JCache implementations. That is when using EHCache at least. The Morphium Internal implementation works
there is a bug with Global Cache override Settings, that are not properly passed on to the underlying Caches
Messaging seems sometimes be affected as well by that. For some reason, we get a mongo exception here and there

category: Computer --> programming --> MongoDB --> morphium

MongoDB Messaging via Morphium

2018-05-06 - Tags: java programming morphium

One of the many advantages of Morphium is the integrate messaging system. This is used for synchronizing the caches in a clustered environment for example. It is part of Morphium for quite some time, it was introduced with one of the first releases.

Messaging uses a sophisticated locking mechanism in order to ensure that messages for one recipient will only get there. Unfortunately this is usually being solved using polling, which means querying the db every now and then. Since Morphium V3.2.0 we can use the OplogMonitor for Messaging. This creates kind of a "Push" for new messages, which means that the DB informs clients about incoming messages.

This reduces load and increases speed. Lets have a look how that works...

Messaging in Morphium - how it works

As mentioned above with V3.2.0 we need to destinguish 2 cases: are we connected to a replicaset (only then there is an oplog the listener could listen to) or not.

no Replicaset => Polling

No replicaset is also true, if you are connected to a sharded cluster via MongoS. Here also messaging uses polling to get the data. Of course, this can be configured. Like, how long should the system pause between polls, should messages be processed one by one or in a bulk...

All comes down to the locking. The algorithm looks like this (you can have a closer look at Messaging.java for mor details):

send a command to mongo, which will lock all messages either send directly to me (= this messaging) or is for all and exclusive (should only be processed once). Every system can be identified with a unique UUID and this id is use for locking, too. it will either lock one or all matching messages - depending if you want to process one or all
read in all locked messages
process them
mark message as processed by me (UUID->processed_by)
do a pause (configured) and go to 1.

Replicaset => OpLogMonitor or ChangeStreamListener

The OplogMonitor is part of Morphium for quite a while now. It uses a TailableCursor on the oplog to get informed about changes. A tailable cursor will stay open, even if thera are no more matching documents. But will send all incoming documents to the client. So the client gets informed about all changes in the system.

With morphium 4.0 we use the change stream instead the oplog to get polling of messages done. This is working as efficient, but does not need admin access.

So why not use a TailableCursor directly on the Msg-Collection then? for several reasons:

it only works with capped collections. Which is not a showstopper in our case, but unpleasant
it only informs about new entries in the collection. But the locking algorithm depends on the update being atomic - hence this is not working. We could try to lock messages by erasing old ones and creating new ones, but this is not atomic and will lead to misreads.

Messaging based on the OplogMonitor looks quite similar to the algorithm above, but the polling simplifies things a bit. on new messages, this happens:

is the incoming message an exclusive one, just try the locking described above. But as we now have the ID, it is a lot simpler and more efficient.
is it non exclusive (and not send by myself), just process it
is it an exclusive message and sent directly to me, process it

usually, when an update on messages comes in, nothing interesting happens. But for being able to reject messages (see below) we just start the locking mechanism to be sure.

how to use Messaging

Well, that is quite simple. Kust create an instance of Messaging and hit start.

Messaging messaging=new Messaging(morphium, 500, true);
messaging.start();

of course, you could instanciate it using spring or something.

Message sending

to send a message, just do:

messaging.storeMessage(new Msg("Testmessage", "A message", "the value - for now", 5000));

this message here does have a ttl (time to live) of 5 secs. The default ttl is 30secs. Older messages will automatically be deleted by mongo.

Messages are broadcast messages by default, meaning, all listeners may process it. if you set the message to be exclusive, only one of the listeners gets the permission to process ist (see locking above).

        Msg m = new Msg();
        m.setExclusive(true);
        m.setName("A message");

this message will only be processed by one recipient!

And the sender does not read his own messages!

Of course, you can send a message directly to a specifiy recipient. This happens automatically when sending answers for example. To send a message to a specific recipient you need to know his UUID. You can get that by messages being sent (sender for example) or you implement some kind of discovery...

        Msg m = new Msg("testmsg1", "The message from M1", "Value");
        m.addRecipient(recipientId);
        m1.storeMessage(m);

storeMessage vs queueMessage

in the integration tests of Morphium both methods are being used. The difference is quite simple: storeMessage stores the message directly do mongodb whereas queueMessage works asynchronously - which might be the better choice when it comes to performance.

receiving messages

just register a Message listener to the messaging system:

           messaging.addMessageListener((messaging, message) -> {
            log.info("Got Message: " + message.toString());
            gotMessage = true;
            return null;
        });

here, messaging is the messaging system and message the message that was sent. This listener returns null, but it could also return a Message, that should be send back as an answer to the sender.

Using the messaging-object, the listener can also publish own messages, which should not be answers or something.

in addition to that, the listener may "reject" a Message by sending a MessageRejectedException - then the message is unlocked so that all clients might use it again (if it was not sent directly to me).

usage of messaging - cache synchronization

Within Morphium the CacheSynchronizer uses Messaging. It needs a messaging system in the constructor.

The implementation of it is not that complicated. The CacheSynchronizer just registers as MorphiumStorageListener, so that it gets informed about all writing accesses (and only then caches need to be syncrhonized).

public class CacheSynchronizer implements MessageListener, MorphiumStorageListener {

}

on write access, it checks if a cached entity is affected and if so, a ClearCachemessage is send using messaging. This message also contains the strategy to use (like, clear whole cache, update the element and so on).

Of course, incoming messages also have to be processed by the CacheSynchronizer. But that is quite simple: if a message comes in, erase the coresponding cache mentioned in the message according to the strategy.

And you might send those clear messages manually by accessing the CacheSynchronizer directly.

And we should mention, that there you could be informed about all cache sync activities using a specific listener interface.

conclusion

the messaging feature of morphium is not well known yet. But it might be used as a simple replacement for full-blown messaging systems and with the new OplogMonitor-Feature it is even better than it ever was.

category: Computer --> programming --> MongoDB --> morphium

New Release Morphium 3.2.0Beta2 - Java Mongo Pojo Mapper

2018-05-02 - Tags: morphium java mongodb mongo POJO

Hi,

a new pre-release of morphium is available now V3.2.0Beta2. This one includes a lot of minor bugfixes and one big feature: Messaging now uses the Oplogmonitor when connected to a Replicaset. This means, no polling anymore and the system gets informed via kind of push!

This is also used for cache synchronization.

Release can be downloaded here: https://github.com/sboesebeck/morphium

The beta is also available on maven central.

This is still Beta, but will be released soon - so stay tuned

category: Computer --> programming --> MongoDB --> morphium

New Release of Morphium V3.1.7

2017-11-21 - Tags: java mongo

We just released V3.1.7 of morphium - caching mongodb pojo layer.

performance increase insert vs upsert
update handling of non-mongoid ID-fields (bugfix)
updated Tests
new strategy for buffered writer: WAIT
setting maxwait / timeout for waitstrategy in bufferedwriter
moving id creation to morphium, implementing proper inserts, fixing bugs
fixing buffered writing on sharded environments
performance increase
mongodb driver version update, checkfornew default fix

Details can be found at the project page on github. You can easily add it to your projects using maven:

 <dependency>
            <groupId>de.caluga</groupId>
            <artifactId>morphium</artifactId>
            <version>3.1.7</version>
 </dependency>

category: Computer --> programming --> MongoDB --> morphium

new release of Morphium 3.1.5

2017-09-29 - Tags: morphium

This release is about tidying up things a bit and includes some minor fixes and tweaks.

fixed some statistics
removing drivers into different project
improving byte array / binary data handling
fixing some tests
fixing checkForNew behaviour, is now a bit easier to understand. If switched off globally, setting it at the annotation does not have an effect.

Available via Maven Central

category: global

git and optware fail after Qnap update to 4.3.3

2017-08-21 - Tags: git qnap storage

for quite some time now, I have a qnap runnin in my basement storing whatever Storage needs, my servers or family members might have.

The qnap is also being used as git server - wich was totally fine the last couple of years but failed recently...

fail after update

I just saw, that there is a firmware update pending for the TS and that this is more or less the last one for this old model (hey, the qnap is not that old, is it 3 years maybe?). There also was a warning message in the release notes that the switch to 64bit might cause some apps not to work anymore...

so far, so uninteresting. The usual blah blah... Unfortunately Optware (which installs addidional opensource software) is only available in 32 bit obviously.

But this is something you will only learn the hard way: the software just woud not work. trying to access the GUI of it, will just result in "internal server error", "page not found" or simply "Permission denied" - depending on what you just tried to make it work.

if you log in via ssh and try to use ipkg in the shell, you will get a file not found error, although you tried to exec the file specifying the absolute path. The linux gurus know - this means some lib is missing!

And in that case, I do not need to dig further - the 32 bit libs are not there anymore.

That would not be the problem, but everything installed by Optware also relies on those libs and hence fails now... bummer!

to solve the problem..

.. you install an alternative to optware called entware. You download the file and install it via Qnap-GUI. Unfortunately this tool does not have a "nice" (the so called gui of Optware was never nice) GUI for this, just a command line command called opkg (mind the o).

after that you only need to create symlinks for the git-binaries (after fiddling with the sshd for enabling pk auth and more than just the admin user):

cd /bin
for i in $(/opt/bin/git*); do
ln -s $i
done

And as always - happy hacking!

category: security

Virenscanner for Mac or Iphone

2017-05-29 - Tags: virus security

originally posted on: https://boesebeck.name

As some of my readers are not that good in reading and understanding German, I'll try to write some of my posts, which might be interesting in english also. I hope everything is understandable so far - This is not a translation, just a rewrite in English. Lets start with the last post about Anti-Virus Software

Anti-Virus software on the Mac or iPhone?

People are more and more concerned about viruses. Also Mac users start to worry about that threat. So, is it neccessary to install anti-virus software on the mac? I was asked that question several times lately...

First of all, this question it totally justified. Everyone should harden his computers and phones as far as he feels safe. Actually, more than a feeling would installing an anti virus software on the mac not produce. As of now there is a handfull of harmful software known for the mac, all of them will be filtered by macs own security mechanisms and thus are not really a thread anymore.

At the moment the Mac is safe - but soon...

"Soon it will be very bad for Mac users. Viruses will come..."

I hear that every year. When the new market share numbers are published and OSX gains. Then everybody tells me, that the marketshare is soon reaching some magic percentage when it will be so interesting for Virus-Programmers to write Viruses for Macs that ther will be a flood of malware. Or will there?

Of course, marketshare is definitely influencing the number of malware for a certain system. But in addition to that, you should take the necessary effort and feasibility into account. And the use... (in terms of malware: what could I gain? Keylogging? Botnet?)

I think, one should take both into account: Is the system easy to hack, it will be hacked, even if almost nobody is using it. Is the systems' marketshare not that high, but relatively simple to hack - it will be hacked! For example: the Microsoft Internet Information Server (IIS) is being attacked far more often than the marketshare leader Apache. When a system is very hard to hack, you need some good incentive to take the effort. Which could be the reason why there is no real virus for Linux or OSX.

And when I write "hacked" its more in a viruses term of use - not remote hacking of user accounts. And: it needs to be done more or less automatically by software. Otherwise there will be no real virus or worm. If somebody wants to hack a certain machine and has the knowledge, he can do it - depending on resource, effort and motivation ;-) I knew a hacker once, you could hire to hack the servers of an competitor for example. Those things are always possible. But this is almost always an administrative problem. There is no real protection against those guys. You can hack any machine you can physically touch - resources and motivation required, of course. Best example: the Jailbreaking of iOS! But if there is enough motivation, resources and knoledge, you're not really safe (see NSA & Co). So it's a question of effort: to hack the machine of a 14 year old student is definitely not as interesting as hacking the machine of a CEO of a big company or a politician.

Same thing is valid for malware and viruses: Malware is not developed for the fun of it (well, at least most of the time it's not). People want to make money with them. This is the only reason why there are Viruses! Maybe that's the reason why there is still the rumor, that actually the Anti-Virus-Software vendors pay some virus developers to spread viruses every once in a while. who knows... i cannot rule that out for sure. I met some Russian guys who claimed that to be true. If so, then I don't understand why there is so few malware for Linux and OSX. That would be a huge market for Anti-Virus-software vendors - millions of users, complete new market segment worth millions or billions of dollar.

I think, viruses are only developed to directly (data theft, credit card fraud etc) or indirectly (by spamming, using hacked machines as bots on the way to the real target, bot nets etc) to MAKE MONEY! And when money is involved: the effort and resources necessary to achieve that must be lower as the estimated revenue of course. So we are at the combination of effort and marketshare again. Marketshare influences the potential revenue (assuming that when more machines are hacked or affected by malware, more money is being made), efforts are the cots. And in some cases this is obviously not a positive figure...

malware in general

First of all, you need to distinguish between the different kinds of malware. In media and the heads of non-IT-guys all malware is named "Virus". But it's necessary to know what kind of software pest is out there in order to be able to protect yourself against those effectively.

The media and in the heads of non IT guys usually every malware is called a "virus". But in order to be able to protect yourself from those malware, it is important to know exactly what you're dealing with. You can classify three different kinds of malware: Viruses, Trojans and Worms - but there are some mixtures of those in the wild, like a virus which spreads like a worm - hence toe umbrella term "malware").

a virus is a little program, which reproduces itself on the system and does dort it's dirty stuff. most of the time, those viruses do exploit some security holes in order to get more privileges. If those privileges are gained, the virus will do things things, you usually do not want him to do - like deleting things, sending data do a server...
a trojan is most similar to a virus, but needs the users help to get installed. Usually it looks like some useful piece of software, a tool of some kind, but in addition to the funktionality you desire, it also installes some malware on the system. Usually the user is being asked, that the software needs more access - on OSX at least. But even if it does not seek privilege escalation, your data still is at risk. See wikipedia
a worm is a piece of malware, that is capable of spreading itself over the network (either locally or over the internet, see wikipedia). You can easily protect yourself against worms if you just unplug the network from your computer (and/or disable WiFi) or at least disable internet access. Sounds insane, but I myself was at some offices and departments, who do exactly that: They are unplugged from the internet in the whole building, only a certain room, which is specially secured, does have internet access - but not into the local network.
a new type of malware just got famouse with wanacry: ransomware these are usually some trojans which do then use bugs in the system to encrypt all data. And you only can decrypt it, if you send a couple of bitcoin to the author.
of course, there are mixrures of all those types. Usually there is a trojan, that acts like a virus on the system to gain root (or admin) access and uses that to spread himself over the network (worm).

on the Mac?

you always get such "warning messages" on the mac, if any malware wants to do something, that is out of the ordenary and does need system privileges. Exactly that happened a couple of months ago when there was a Trojan, who was installed using Java and a security issue therein. But still, the users were asked, that the software needs more privileges. And enough people just said "yes" to very question...

Please do not get me wrong, I do not want to deemphasize malware. It is out there, and does cause a lot of harm and costs. But you can be saved by trojans more or less by using common sense:

Why does the new calculator app need access to my contacts?
Why does my new notes app need admin permissions?
why does software XY ask about this or that permission?
is it clever to download tools from an untrusted source, especially if this source does offer cracks or exploits or something?

It is getting harder, if the trojan uses its newly gained privileges to hack the system itself, maybe even exploiting additional security issues there, so that the user is not being asked. Then a secure operating system architecture is helping to avoid those kind of things. Which is usually implemented by all unix OS.

Viruses and worms can not be avoided so easily hence those do exploit bugs in the system. But even then, Unix based systems are a bit better suited for that case than others.

This is according to a very strict separation between "System" and "Userprocesses" and between the users themselves. And, especially on OSX, we have Sandboxing as an additional means against those malwares. And the graphical user interface is not bound so tightly to the operating system kernel like it is in Windows NT for example.

But, overall, the Admin of the system is the one, really determining how secure a system is. He should know about the problems, his OS has and can take counter measures accordingly.

Malware on mobile devices

if we are talking about malware, whe should also have a closer look at mobile devices. Especially Smartphones and alike are often attaced, because they do have a lot of interesting data which are just worth a lot of money. Or you can just make money directly (e.g. by sending expensive SMS).

to "beak into" such a closed system, very often security relevant bugs are exploited. But sometimes just social engineering is also successful.

Usualy the user is than made to do some certain action, that does involve downloading something, that is installing a trojan on the system. or just opening the system that the attacer than can install some malware. Or you just "replace" an official app in the corresponding appstore.

Trojans on the smartphone usualy are masked as litte useful tools, like a flashlight app. But they then copy the addressbook and send out expensive short text messages, switch on video and audio for surveillance and so on.

It's hard to actually do something against that, because you do not know, ob the app, you install does something evil or not. Apple is trying to address this problem with the mandatory review process that all apps in the Appstore need to pass. All apps need to pass an automated and a manual check before anyone can download it. The apps are for example not allowed to use unofficial API (for accessing the internals of the os) and that the app does exactly what the description of the app tells the users it does.

This is no 100% protecion, but it is quite good (at least, i do not know any malware on the appstore right now).

But I would also name WhatsApp, Viber and alike as malware. Those do exaclty that, what a trojan would do. Grab data, upload them to a server. But here the user happily agrees and likes it.... but that is a different topic.

on iOS users are a bit more secure, than on andriod (if you do not jailbreak your iphone). Android is based on Unix, but some of the security mechanisms within uinx have bin "twisted". So there is a "kind of" Sandbox, just by creating a new user for every app on the device. So all processes are separated from each other. Sounds like a plan. But then you end up having problems with access to shared resources, like the SD-Card. This needs to be global readable!

Also the Security settings of apps can at the moment only take "all or nothing" (that did change in later versions, at least a bit). So you can either grant the app all the permissions, it wants. or No permission at all.

Problematic is, you need to set the permissions before actually using it. This makes it very easy for malware programmers, as people are used to just allow everything the app needs.

IN addition to that, Andriod apps do have an option to download code over the internet - this is forbidden in iOS. And there is a reason for it: How should any reviewer find out, that the code downloaded stays the same after the review? Today I download weather data, tomorrow some malware wich sends chareable short texts?

Another problem is, that there is not one single store for android but more like a quadrillion of them. Hence you can install software from almost any source onto your andriod device.

of course, every os does have bugs which might be used to execute good or evil code on the device. Hence there are updates on those OS on a regular basis, which should fix security relevant bugs and issues. with iOS you can be sure, that you get updates for your device and the OS on that for at least a couple of years. (current iOS run on 3 to 4 year old hardware still). With android it is not as easi to make such a statement as the support is strongly depending on the vendor. It might be, that support for devices not older than 1,5 years are stopped. Especially the cheap Android phones loos support quite ealry, which means there are still Android 2.x out there (and you actually still can buy new devices with that installed). Including all the bugs, that the old OS version had - which makes it quit interesting for malware authors.

in combination with the a bit more insecure system and the unsecure sources of software, this makes android a lot more prone to be hacked or infected by malware. And this makes it especially interesting for the bad guys out there.

This is leading to really rediculous things like virus scanners and firewalls for smartphones. read it here in German

You can say about apple, what you want, but the approach of the review of every app for the appstore is at least for the enduser a good thing (and by that I do not mean the power user who wants to have his own Version of the winterboard installed). Even if you are not allowed to do "all" with your phone - Normal users usually do not need to.

And the poweruser can jailbreak his iphone still - and if he knows what he is doing, it might be ok.

installed software as a gateway for malware

Unfortunately viruses, trojans or more generic malware, can use any bug of any software on the system, no matter if it is part of the OS or not. So a breach can be done via a 3rd party software. Like the "virus" that was infecting a couple of thousand macs through a installed java. In this case, the user again was asked several times(!) if he wants to grant admin permission to a java app - if you agree to that, your system is infected. If not - well, nothing happens. Common sense is a great "Intrusion Prevention System" here.

Of course, osx or any other operating system cannot avoid 3rd party software of doing some dubious things - especially, if the user agreed to it. But the software is only able to gain the permissions, what the software that was used as gateway has. An on OSX and iOS all applications run in a Sandbox with very limited permissions. If the app, a malware uses as gateway does not have admin permissions, well, the malware won't have it neither.

If all 3rd party software you run on your system only has minimal permissions, then a malware that would use those as a gateway would also have minimal permissions, and could not do too much harm (and could easily be removed).

But the thing is, just getting access as a normal user is not the goal of such a virus vendor - they want your machine to be part of a botnet in order to sell your computing power or to use it in the next DDOS-attack. Or just use it as spambot.

Also it is in the best interest of this virus vendor to make it as hard as possible to remove the software from the system. So everything needs to be burried deeply into the system files, where normaly no user takes a closer look at.

And this is usually only possible, if the malware would get admin permissions. It could use "privilege escalation" hacks in order to gain more permissions - best case, without the user knowing.

Usually, the user should be asked, if any process tries to gain more permissions, and the user may or may not agree to that (that happens every time, a process tires to do something outside of the sandbox). of course, that would be bad, as it would reduce the success of the virus. So virus vendors try a lot to avoid this kind of informing or asking the user.

on unix systems this is quite some hard task, or at least a lot harder as on windows OS see here or here. In almost all of the cases, on osx the user is informed about software that does do something strange.

But there is one thing, we should think about even more: if any software could be used as a gateway, I should reduce the number of programs on it to a minimum (especially those, with network functionality... which is almost any app nowadays). Especially I should keep software that runs with admin permissions do the absolute minimum - which is 0! Unfortunately, virus scanners and firewalls and such "security" software, need admin permissions to do their job. This is one of the reasons, why anti virus software is very often target of attacks from malware and viruses and end up as spreading the very thing they try to protect us from. (this has happened on windows machines)

Then, count in that a Anti-Virus software can only detect viruses, that are publicly known for a while, you actually would not increase the protection a lot by installing this on your machine.

Same thing goes for firewalls, which have their use on windows systems unfortunately, but not on unixes or osx. How come?

Well, on unix systems the network services are usually disabled, or not installed! so the visible footprint on the internet for such a machine is quite low.

Windows on the other hand, is depending on some network services to run, even if you do not actively use it. Disabling those serivces (and SMB is one of them - this was used by wannacry!) would affect the system in a bad way and some things would not run as expected see here.

Hence, if your system does have a minimal footprint - or attackable surface - you do not need a firewall.

Btw: do not mix up this local firewall, with a real IP-filter firewall that is installed in routers!

Virus scanners on servers

So, there is a lot that explains, why using virus scanners on the desktop (especially if it is a unix desktop) can have negative effects or at least no effect. So, you're probably fine without them...

But on servers, things look a bit different.

If i have clients are not well maintained or I just do not know (or just windows ), I want to avoid storing data on my server, that could infect them. So, even if the viruses do not infect my server, or my mac. The mails could be read by other clients, that might then be infected. So, be nice to your neighbors...

Do not forget, virus scanners do need some resources. And sometimes a lot of it (they monitor every access to/from the system, which in return can or will slow it down to a certain extend).

Security is not for free

Whatever you do, security comes with a cost. in "best" case, things get inconvenient to use, cause you need to do complex authentications or need to agree to a lot of popups that pop up every second (remember Windows Vista? )

in the worst case, there are errors because of the high complexity, or expensive bacause you need additional hardware (iris scanner, external firewalls, Application-level firewalls that scan data for viruses...) and still being inconvenient at the same time. And time consuming (those systems need to me maintained).

So, you need to decide, what level of security do you want, and what is senseable. The use of an Iris Scanner for the Bathroom is probably a bit over the top... don't you think?

common sense

the best weapon in our hands against malware still is the thing between the ears! Use it when surfing, when installing software. No software will ever be able to stop you from doing something stupid to your system.

So, it is not ok to feel to safe when being on a mac. This leads to sloppiness! Passwords for example, need to be real passwords. If the password could easily be guessed, why should a malware take the detour for hacking the system? It could just "enter" it and you lost your system to the bad guys....

I don't want you to get paranoid on that neither! Just keep your eyes open. When installing software, only do it from trusted sources. And, from time to time, have a closer look. There was malware available in the AppStore for a couple of days / weeks before apple removed it. Even the best system can be outwitted.

You should think about, which apps you use and which not. And even apps, that are not really malware per se, dan do harmful things - like whatsapp and viber. You should ask what is happening there! I mean, whatsapp is uploading the addressbook to facebooks servers and the people whos data you upload there, are not asked if they like that... just a small example...

Just remember: if the product is for free, then YOU are the product

There is no such thing as free beer!

conclusion

I tried to be not tooo aniti microsoft - which is hard, because most of the security issues are only existing on windows systems. Unfortunately on windows the user needs to make it secure and stop it from doing harmful things.

Anti Virus software does lull in the user to make him feel safe, but most of them really have a louse detection rate. And really new viruses are not detected at all.

So, should you install anti virus software on a mac? You need to decide yourself, but I tend to "no, you should not". But there are valid reasons to see it differently. But I am not alone with my thoughts: see here and here.

But you definitely should distinguish between desktop and server, as you may be serving out data to windows machines as well, a virus scanner might be a useful thing.

Almost all I wrote here is valid for osx and for linux or other unixes. Right now, there is no know wide spread malware out for unix based systems, that I know of.

Additional links

category: Computer --> programming

Markdown - THE cool way of writing text?

2017-05-26 - Tags: markdown

What's that?

We do write a lot of texts every day. Everyone of us, who is working with computers. And we all struggle with formatting texts. Text without proper indentations, emphasizing or boldface is a bit boring to read and you lack an additional means of expression.

Most editors, even in web nowadays support "WYSIWYG": What you see is what you get. This is looking nice, most of the time it acutally works. But is hard to use (you switch between mouse and keyboard very often). Well, it annoyed me at least. Especially if you edit text, and after editing it, the italics are not at the right place anymore and vice versa.

So, why not put special sequences in normal text an have it rendered afterwords?

This is not really a "new" idea: Not so long ago, this was one of the prefered ways of formatting Text. You enrich standard text with so called markup codes to make it possible to define special formats and what not. Most of you probably use this but do not realize that: HTML is the most pupular version of a Markup Language. Of course, nowadays you see it in your browser, and not the code. So it became a lot more, than just "Hypertext".

One other well known (well, if you are a nerd ) implementation of a markup language ist LaTeX - this is a markup language especially concentrating on typesetting. So you will get a very good looking printout (but can use your favorite Text editor). Maybe that is one of the reasons, why it is so popular when using for master thesis or diplomas.

Those "languages" are a bit too complicated and complex to just create an email or so. And that is, where Markdown comes to help.

why that?

But why would you want to have your text enriched with command sequences, if you can do it with the mouse cursor?

Yes, it works using a mouse. But there are people amongst us, which type so quickly that switching between mouse and keyboard actually slows them down. And I am one of them.

And the beauty of markdown is, that the sequences being used for formating text, are easy to reach more or less standard characters (nothing special).

For Example: If I want something to be amphasized (italics), I just put an underscore _ before and after the sequence - done!

or I want to have a numbered list... I start a paragraph with a 1. - this will be indeted as expected and als subsequent lines accordingly.

But there is a lot more, a list of what is possible can be viewed here.

I for instance write everything here in the blog with Markdown! So I did not have to use a WYSIWYG-Editor for the Frontend. And texts are stored every "simple" and can be indexed quite easily. So no proprietary XML-Stuff or even worse some binary format.

Especially if you are a developer, Mardown does have advantages. If it is properly configured, your sourcecode (which are usually part of the documentation) can be highlighted (if markdown is configured properly):

Example Java Code

public static void main(String args[]){
System.out.println("Markdown is colorful");
}

Example Bash-Shell script

#!/bin/bash
echo "and understands different languages"

for i in $(ls); do
   echo "This is rocking $i"
done

I will add another post here some time, because in order to get the syntax highlighting to work with the markdown renderer library I use, I had to extend the software a bit.

cool, and now?

Well, if you want to use Markdown, you will have to learn those "commands" or Sequences. But, it is really worth it, to do. Especially if you are a touch typist and fast at it

There is a lot of tools to easily create stunning texts. And there is a huge community for markdown, who do come up with new features, new tools. So there is an extension fro Markdown like CriticsMarkup, or even board the exensions from multimarkdown.

All in all , this is a really powerful toolset which helps you concentrate about the thing that really matters: The Text!

here you can see a list of all standard functionalities in markdown.

Markdown on the Mac

on the mac there is already a list of good tools that support markdown or help with syntax highlighting. Those editors sometimes have a live preview and are able to export your text as HTML, RTF or PDF. I will create a couple of Test of Tools posts for those editors. Hier a little list about some tools I used:

MultiMarkdown Composer
[MacDown)(http://macdown.uranusjr.com)
BBEdit, Textwrangler and Sublime (syntax highlighting, no real "Rendering"
iA Writer (in the Appstore)
markdown-here

Also most IDEs do support Markdown (Xcode, IntelliJ & Co). So, to create documentation in your Project directly, markdown definitely is an option.

Unfortunately the support in mail is a bit of a problem now. Right now, apple Mail does not support markdown. But you can use markdown-here to help minder the pain.

Of course, that is a bit inconvinient to use. Better use Mailclients that support markdown natively like MailMate or AirMail2 (the latter one does have some severy data privacy issues, but that is a different topic).

verdict

Markdown does have a lot of advantages especially because of the excellent tools available already. You just type your text, concentrate on typing. Formatting is done automagically. Afterwards you can export it as PDF, RTF or whatever.

So markdown definitely is an alternative not only for developers, but also for users who create a lot of texts like books, reports, emails and what not.

But you will have to get used to the tools, and you need to add the rendering of the document to your time sheet as well....

category: global

ErgodoxLayoutGenerator

2017-05-20 - Tags: english ergodox-ez

I was a little bit bored of always creating the layout by "hand" and I have to admit that I never got the Massdrop Configurator to work properly. And I never managed it to get the Overview PNG up to date with all of my changes. Hence I decided to create a little tool, to help me with that.

it should create the keymap.c file
it might also read it - no need for an own filetype
it should have support for Macros - there are only a couple of macro types that are useful. Like type that sequence of keys or if pressed, press these keys, if typed, type this sequence
it should create a GUI that could be used as documentation for it

So... The first milestone is finished, I created a little java application that can read my keymap.c and is able to show the layout graphically. Yes I know, it does not look that good, but it is ok...

Here is the first Screenshot of this simple tool, showing my current osx_de layout:

Things still to be done:

proper C-file export
macro support
sorting of layers
KeyMapping
UI improvements...

so for now, this tool actually helps with documenting of keymaps. It reads in a keymap.c file and shows it in a more graphical way. This is an example:

The tool is available on Github: https://github.com/sboesebeck/ErgodoxLayoutGenerator

Disclaimer: This is in prototype phase. Not really more than a proof of concept. So use it at own risk. Same for the code - it works, but there are some ugly parts in it...

Here is the documentation PNG of all layers put together. Creation of this took 5 mins.

update

Although some people thought this would be an April's fool prank - it isn't. This tool really exists and really works! It is now capable of creating overview PNGs with a click of a button. This is the overview PNG for my own layout osx_de:

But it works with all other layouts so far as well. Like the default one:

This is an example, where the parsing worked fine, but the file lacks some information. The layers do not have descriptive names. And there you also see that there are a lot of macros being called. Here most of them are just unicode output of special letters, but the ELG does not show them properly:

Things still to be done:

create a release, executable jar file, so that everybody can just test it.
fix didplay of keys - EXCLM should be a !.
fix parsing of macros. especially the dealing with unicode keys
if the display is correct, deal further with the input of keycodes
store as proper keymap.c

First BETA Release available Go have a look here here. This is a BETA Release, not all functionality is implemented yet. But you can create your documentation overview PNG file...

should run on all machines having java installed (current JDK8!)

category: global

Tweet: got the first version of editing ready. Only Macro...

2017-05-20 - Tags: tweet

got the first version of editing ready. Only Macros missing for now. Next: keymap.c creation... https://t.co/Ftc4bUl5Fp

category: global

Tweet: New documentation PNG for the @ErgoDoxEZ layout os...

2017-05-20 - Tags: tweet

New documentation PNG for the @ErgoDoxEZ layout osx_de. was super easy to create now... https://t.co/UsOtVoEYdk

category: Computer --> programming --> Java

ErgodoxLayoutGenerator Documentation

2017-05-20 - Tags: english ergodox ergodox-ez java-2

If you read my blog, you might have noticed, that I'm fond of cool keyboards. We IT-guys use them the whole day, but most keyboards are just awful to work with. So I'm glad I found a "proper" ergonomic one, my ErgodoxEZ (look at http://ergodox-ez.com for more information or read my review here).

One of the greatest things about the ErgodoxEZ is its programmability. But you actually need to know how to code, in order to get it to run easily. And even if you do, it is not very intuitive to create a c-program that runs on your keyboard and is showing your layout.

There is a WYSIWYG-Editor at massdrop.com, but unfortunately I never got it to work properly - and it is somewhat limited in functionality (like Macro support etc).

Hence, I started creating my own little tool for creating my layouts and optimizing them...

The ErgodoxLayoutGenerator

The ErgodoxLayoutGenerator (very clumsy name, but I could not think of something more better for now - lets just call it ELG for short) is programmed in Java, so it should work on all machines and OSs java is available for. You need to have the latest version of the Java8 installed... As I got some feedback already: It needs to be the latest official Oracle JDK or JRE, it does not work with OpenJDK out of the box. If you need any help using OpenJDK, just contact me...

The idea is pretty much similar to the one in massdrop, but the ErgodoxLayoutGenerator is built around the qmk firmware, and it generates a keymap.c file for your local installation!

Prerequisites and remarks

you need a current version of Oracle Java installed on your system
you should have a current version of the QMK-Firmware repository cloned to your machine for the LayoutGenerator to make any sense. It will still generate keymap.c files for you, but you cannot flash them to your keyboard.
Although the qmk-firmware is available for a bunch of keyboards, the ErgodoxLayoutGenerator is only covering the Ergodox and ErgodoxEZ (hence the name).
keep in mind, that the ErgodoxLayoutGenerator is still in development and was not heavily tested on different environments or OS's.
If you have any feature requests or ideas you want to share, please visit the github project page here
The ELG cannot reach the flexibility of a c-program of course. It is limited to the most basic functionalities. For example it lacks support for UTF-Characters and the GUI definitely needs to be refurbished. But I never wanted to win a desgin price with that - it should just work.
The ELG does read the keymap.c file and parses it to some extend. But that means, it is very relying on the structure of the file to be more or less similar to the "official" ergodox layouts that come with the qmk-firmware. If you want to use the ELG to work with your layout, you should make sure, that your keymap file does follow this lead
There is support for custom macros, where you can just add c-code to your layout. There is no checking if that is correct or highlighting or whatever.
The support for non-us keyboard layouts is a bit... clumsy. You can use the proper keycodes in most cases, but especially when creating macros, the automatic generation of macro strings, is not working. It will create macros for the US layout. This is limited due to the fact that there is no regulation on how to name the keycode definitions. For example, there could be a KC_ACUT, DE_ACCENT, DE_OSX_ACT all "meaning" the same key.
As the ELG is reading in your keymap.c file, things might get messy. If you use one of the standard keymaps, all should go fine. But if you use some of the advanced stuff, things will go wrong. It will probably parse the keymap and most of the functions will show up fine, but some things might go missing. At the moment there is no support for the FN[1-9]-keys! So, if your keyboard uses those, please make sure, that you replace the functionality with a macro.

The parsing of the C-Files does have its drawbacks, but the great advantage is the possibility to have ELG read in existing keymaps! That does work most of the time.

Getting Started

Usually it should be ok to get the latest release from the gitub page. Download the JAR-File attached to the release and double click it. If java is installed properly, it should start up fine and you should see a screen with an empty layout:

starting it from commandline

If the above does not work, you can try to run the jar file from commandline with java -jar ergodoxgenerator-1.0BETA2.jar - or whatever file you downloaded. If it still does not work, you'd get a proper errormessage then. On the github page you can create an issue for that.

Sometimes it might be helping, to not use the JAR-Start funktionality of java, but run it manually: java -cp ergodoxgenerator-1.0BETA2.jar de.caluga.ergodox.Main. If you get the same error as above, please create an issue at github.

compiling it yourself

If you're a java-guy, you can compile it yourself (and hopefully contribute to the project). The project is a standard maven project. So if you cloned the repository to your local machine, running mvn install should compile everything. Your executeable will then be in the directory target and called ergodoxgenerator-1.0-SNAPSHOT-shaded.jar. This one should be executeable...

Defining the QMK-Sourcedir

At the bottom of the main window, there is a button called set qmk sourcedir. Here you should set the root directory of the qmk sources. This is necessary for putting the layout at the proper position at the end. If you did not specify this directory, you always need to navigate there manually.

Opening a keymap

If you defined the QMK-Sourcedir, the open dialog will start in the correct directory for the ergodox layouts. You choose the directory of the keymap, as all keymap files actually are called keymap.c.

When the keymap was parsed successfully you should get a display of the base layer of this layout.

UI explanation

Layer chooser

Usually an ergodox layout consists of several different layers. Like when hitting ALT on a keyboard, all keys do something else. But here you are more or less free to define as many layers as you want (not really, your keyboard has limited memory). To switch to the different layers, you need to press or hold a key (see below). when changing the layer in this combobox, the layout will be shown accordingly.

Adding, renaming, deleting layers

When creating a layer, you only have one layer called base defined. The buttons on the top let you create new layers, rename them or delete them. Attention: Deleting layers and still having layer toggles or macros referencing them, will cause unexpected behavior. Also you should not rename the base layer, as this might also cause problems later.

LED indicators

on the top right of the window, you can see the 3 LEDs the ergodox does have. You can switch them on and off by clicking for the selected layer. This reflects the behaviour of the LEDs when flashed on your keyboard.

the keys

the main portion of the screen is filled with keys. These represent the corresponding key on the ergodox keyobard. If you select a key, it will be marked (green border) and a more detailed description of the key is shown in the lower part of the window. Assigning functionality to a specific key can be done via the context menu. Just right click on a key, it will be marked and then you can

clear: well, this usually means, that the keycode KC_TRNS is assigned to that key. This code states, that this key should behave as defined in the "previous" layer (usually base). This can of course not work in the base layer!
assign a key: well, you assign a keycode to that specific key. The keycode is represented in Text. All keycodes starting with KC_ are the "default" ones. There are also different keycodes for different locales or OS, like DE_OSX_. You can also assign here a combination of keys. Like "Shift-A" or "CMD-S". If you specify more than one modifier, a macro will be created for you!
assign layertoggle: This will create a key that can toggle a certain layer on and off.
assign layertoggle/type: this is temporarily toggling to a specific layer as long as the corresponding key is held. When the key is released, you return to the layer before. If you just type that key, it will issue a different keycode. For Example, y does issue CTRL when held in my layout
Assign macro: This is the most feature rich thing, see below. Attention: The macro functionality is not 100% implemented yet!

the legend

At the lower part of the window there are representations of some colors and keys. Those state, that a green marked key would be of type Layertoggle / type, and hence shows two informations: first line is the key being typed, 2nd line is the layer to switch to as long as the key is held

Save Button

There you will be asked for the file to store the keymap to. This file should always be named keymap.c and should be stored in the QMK-Sourcedir at keyboards/ergodox-ez/keymaps/YOUR_KEYMAP, where YOUR_KEYMAP needs to be replaced with the name of your keymap.

When you want to store a completely new keymap, you need to create this directory yourself. You can do that from within the save dialog.

Save Img

This will create a PNG showing all layers. This is useful to add to your layouts, if you want to publish them and have them merged to the official qmk repository as it makes it easier for others to use your layout. like this one:

open

Open a keymap. You need to choose the directory, not the file!

reopen last

If you are a bit like me, you usually work on your own layout again and again. The button "reopen last" will open the file you last opened or saved!

create

Creates a completely new layout - Attention There is no "are you sure"-Question yet! IF you hit that button now, you'll end up with a new empty layout!

Assigning keys

When assigning keys, you first need to choose the "prefix" of the keycode names. Usually the prefix is related to the locale. Like "DE_OSX" is the German OSX version of some keycodes. all keycodes starting with "KC_" are the default (US-layout) keycodes.

You can add a modifier to the key if you want. And there are 2 different ways these modifiers might work: all at once (like SHIFT-A for a capital A) or the modifier when held, the key when typed! Like when holding the key Y, you hold CTRL, when typing it, it is just a plain old y.

assigning a layer toggle

For this functionality you only need to define the layer you want to switch to. Quite simple. When flashed to your keyboard, the corresponding key will switch on a specific layer when hit, and switch it off again when hit again. If you switch to such a layer in that way, it is probably a good idea, to set the leds properly.

assigning layer toggle / type

as already mentioned above, this will create a key, that will temporarily switch to a layer as long as the key is pressed. If you only type the key (= pressing it shortly), you will just type a normal key.

assigning a macro

The assignment of macros is quite easy, you can just choose one from the dropdown and then hit "assign macro". This works only like that, if the file you opened has some macros defined.

If you hit the "new Macro" or "edit macro" button, the Macro editor is shown. You can create, delete, or edit the macros in this layout.

The ErgodoxLayoutGenerator supports these kind of macros:

TypeMacro: This is a typing macro. Which means, it will send a series of keystrokes when the key is pressed
HoldKeyMacro: this is very similar to the above, but there is a different set of actions that can be defined, when the key is released. Example: When the key is pressed, CMD+SHIFT is pressed, A is typed. When released, CMD+Shift are also released.
LongPressTypeMacro: Different behavior is the key is pressed and held or only typed shortly
layer toggle macro: well, use a macro to toggle a layer
Custom macro: custom c-code

ATTENTION The Macros only support keycodes that do not represent a combination of keys. For example the keycode DE_OSX_QUOT is actually a replacement for LSFT(DE_OSX_HASH). This will not work in a macro, it will only send the keycode DE_OSX_HASH without the modifier. If you want your macro to work in your locale you need to be aware if this key is typed with a modifier or not.

All actions a macro can do, are the following:

DOWN(KEYCODE). press down a key
UP(KEYCODE). release the key
TYPE(KEYCODE). Type this key
W(199) . Wait some milliseconds, 199 in this case
I(1). Change the Interval and set it to 1 in this case

you just add these together, separated by comma, and you have your macro actions for the specified case.

Additional notes

This little project was first of all only built to run on my machine and make it more easy for myself to tweak with the layout. So it is only tested on a Mac OSX machine, not sure how it will work on windows or linux.

There are still a lot of things missing:

gui layout could be improved a lot. Especially in the Macro-Workflow
Errorhandling is completely missing
some "are you sure"-Dialogs need to be added.
store if you changed something on the layout, if so, ask before quitting, erasing...
code quality is... prototype grade!

Compiling it

the latest versions of the ELG do have a "compile" button. When you have your keymap saved and the qmk-sourcedir is set, you can compile it. This is done by running the commands make clean and make in the qmk directory of the ergodox-ez.

This can only work, if your system is capable of compiling it. Please ensure that you have everything installed and is in your path. Take a look at the qmk-github page for more information on how to prepare your system.

When the compilation finished successfully, you can read the log output. USually that is not very interesting, if everything worked fine. On errors you can check closely what went wrong.

When this dialog is dismissed, you will be asked if you want to copy the .hex file - which is the result of the compilation - to the keymap directory. This is useful, if you want to submit your code to the official github project. When your keymap does also have a .hex file, everybody can just download an use it without having to deal with compilation and stuff.

If you just compile for yourself, hit "no" there...

little Update:

The release Candidate is here... completely with support for a new Type of Macro called ToggleLayerAndHold which will toggle a layer as long as the key is pressed, or, if the key is only typed, toggle the layer as with the TG() function call.

Also, the latest version will add a list of all Macros to the PNG file and a short description... helpful for documenting things.

And now there is a compile button, which will compile your layout if everything is set correctly! The resulting .hex file can then be uploaded to your Ergodox or Ergodox-EZ!

Bugs and issues

If you find anything not properly working and you think it is a bug in the ELG, do not hesitate and create an issue at the github page. Please provide the following:

the log output (for compiling and such). If it is related to the ELG, the log is not shown by default. If you want to see the output, you need to start the ELG in the shell / commandline using java -jar ergodoxgenerator-VERSION.jar
what happened
what did you want to happen
what did you try to accomplish
maybe the resulting keymap

category: global

jblog-downtimes

2017-05-20 - Tags: jblog

hi ho,

as you all know, the software here is quite new and of course i found some bugs i did not realize till I went live...

unfortunately I cannot deploy without downtime yet sorry

I will try to keep it to a minimum!

Thanks!

category: Computer --> programming --> Java

Jblog - Java Blogging Software

2017-05-20 - Tags: java blog

I explained here why PHP and Wordpress is a pain in the ass and that I decided to build a new software on a stack I knew.

As I am the only user of this software, I did not have to care about Mulit-User-Role-Permission Management, or Theming or plugins. Just a straight forward webapplication. But it should to it in the following:

java as a basis. This language I speak best
Freemarker as Templating-Engine - have a lot of great good examples with that
It should do everything based on markdown. Wordpress never did that in a proper way. It was always a hassle. I use Flexmark - available at github.
mongodb as Storage
and of course Morphium as POJO Mapper (download it here)
Spring Web and Spring MVC
multilanguage support (well, 2 , I only speak english and German)
simple whitelabeling

Well... er.... that’s it. You see, quite simple actually. I used Bootstrap 4 in the frontend although it is not officially out yet. But I did think, by the time I finished this blog software, bootstrap 4 would be out... ok...

The application runs within a tomcat, called by an NGinx which also does SSL-Termination. All more or less standard.

Of course there will be changes over time. but for the first try it is not too bad...

category: Computer --> programming --> Java

New Version of Morphium available V3.1.3

2017-05-17 - Tags: java morphium mongodb

This release contains some minor fixes and improvements:

fixing some testcases, adding new tests
bug in storelist fixed, where it does not honor the disableBufferedWriterForThread
improving aggregator, making it more easiy to use (no need to call end() on group anymore)
adding aggegator functions $stdDevPop and $stdDevSamp
caching fix for ID-Cache, cache projection fix
avoiding ConcurrentModificationException in whe flushing buffered writer
minor improvements to performance...

You can either get it from github or via maven central:

    <dependency>
	<groupId>de.caluga</groupId>
	<artifactId>morphium</artifactId>
	<version>3.1.3</version>
    </dependency>

category: Computer

New blogging software

2017-05-16 - Tags: java jblog security

originally posted on: https://boesebeck.name

I did complain about wordpress several times (for example here). I took that for an opportunity, to take on my software development skills and use a weekend or two to build a new blogging software. Well, th result is this wonderful (well... hop so) page here.

PHP sucks

To stop all PHP fainbois from whyning, I do not like PHP very much, because I don't know it very much. Hence, wordpress is also kind of a mystery for me. The configuration works with luck, let alone get php to do what you want in a more secure way.

so, my blog was hacked several times during the last year now and this is pissing me off! So, I wanted to use a java based solution, but it seems like there is no simple, easy to use one out there.

so why not do it yourself?

exactly. That was my thought also. Could not be so complicated, could it? So, I wanted to create a blogging software that

has a simple technology stack
does not need a complex plugin funktionality. If it cannot do, what I like it to do, i rewrite it
themes or designs... well... er... could be better, but I think this is ok
Security, that is the point. I created the blogging software (called it jblog - not rally creative) myself and it is not so complex as wordpress. So we should be ok. I guess. But I know for sure, that th standard wordpress exploits wont work no more!
Intrnationalization... also a topic. jblog does only do 2 languages, German and English (I do not speak more, so I don't need more for my blogs).
whitelabeling. I have a couple of domains, I wanted to reuse / revive with this project.
one administration: I did not want to create the same thing 3 times, I wanted to have the same thing look like 3 different hings. Hence there should only be one administration page.

jblog

I am quite ok with what I accomplished here. Although it took longer than one weekend, it was finished quite fast. I lik that.

But please: if some links do not work anymore, some images look strange or are missing - I will fix this eventually

the different blogs

boesebeck.name - this blog here

the private main blog. Will cover topics like hobby, drones, games, gadgets etc.

caluga.de - the java blog

There I will put all my opnsource stuff, like morphium. And all the other programming tips and tricks I wrote over time. Hmm... seems like 'java blog' is not the right term...

boesebeck.biz

This should be a business site anyways. So, here I will put in topics about my professional carreer, Scrum, processes etc.

Updates

well, this is going to be tough. I cannot produce content for 3 full blogs. Even filling one is quite hard. But I will try. And we will see, how that works

technical discussion

as mentioned above - not here, but at caluga.de

category: Java --> programming --> Computer