The whole thing starts when I wanted to change the default font in Mathematica (the default is buggy on my Linux box if ms-fonts are not installed). Later I also modified some other defaults. It's not very trivial thus let me note it down the steps:
(1) Open a notebook, in the menu choose "Format" -> "Edit Stylesheet".
(2) Choose a style to change. For example, "Text", in the menu at the top left.
(3) Modify fonts, etc.
(4) Save this stylesheet to a .nb file. Optionally one can also click Install StyleSheet so that the stylesheet can be found at "Format" -> "Stylesheet..." menu.
(5) Open menu "Format" -> "Option Inspector", make sure "Global Preference" is selected (where the default is "Selection") in the popup menu.
(6) Search for DefaultStyleDefinitions, change it to the saved stylesheet file. Click apply.
Close Mathematica and launch it again. The default font is changed.
Another change I made is added an opener for a group of cells. This can be done at steps (4). Here, search ShowGroupOpener and set it to True, and save. I also changed the color so that it does not look annoying.