> fiandes.io

Notes on self-hosting essential apps

2016-09-14T00:00:00+00:00

These notes contain a relatively simple guide to set-up a fresh Ubuntu server and self-host a number of (what I consider) essential apps.

There are plenty of reasons you want to self-host applications (privacy, who owns your data, your data being stolen, etc.). On the other hand, self-hosting forces you to be very proactive about your data, chiefly:

establish a rock-solid backup policy
keep your server secure

Both back-up and security are not easy feat to get right, so self-hosting requires a certain degree of discipline. Having said that, with a relatively small time-investment, I have never suffered any major (or minor) data-loss or data breach.

I started to look for a self-hosted alternative to Dropbox back in 2013, mostly out of privacy concerns, and I continued to add more self-hosted applications to my server (~~a 4GB RAM box hosted with Hetzner~~). At this point (end of 2016), my digital backbone is almost completely self-hosted, except for an e-mail server, which I happily leave to Fastmail.

The configuration process described here is based on Ubuntu 16.04 LTS and does install and configure the following applications:

Resilio Sync (formerly known as BitTorrent Sync)
Baïkal Baïkal (Cal and CardDAV server)
Tiny Tiny RSS
DokuWiki
Piwigo (photo gallery)

All the web-facing applications use HTTPS (thanks Let’s Encrypt!).

It is assumed that the server IP address is associated to a DNS entry and that all the applications have a sub domain, like so:

- btsync.mydomain.com
- dav.mydomain.com
- wiki.mydomain.com

… and so on.

There is an incredible number of self-hosted applications which can be used to replace closed-source, paid applications. A good starting point is this ‘awesome’ list and this sub-reddit.

OS

I assume that Ubuntu 16.04 LTS is installed on a new server and the logged in user is root.

The very first commands to run will update the package repositories and the system.

apt update && sudo apt upgrade

Change the root password:

passw

Setup a login user, with sudo permission

adduser youruser
gpasswd -a youruser sudo

Hardening

I don’t want to be hacked, rooted or my server to turn into a remotely controlled zombie. Harden.Your.Server. I have been using this excellent collection of security recomendations to harden my box, and you should do the same.

The crucial bits are documented here. I warmly recommend to read the above guide and decide for yourself how paranoid you want to be when hardening your server.

Firewall

# install UFW (Uncomplicated Firewall)
sudo apt-get install ufw

# allow ssh and apache 80/443
sudo ufw allow ssh
sudo ufw allow "Apache Full"

# enable the firewall
sudo ufw enable

Harden SSH

Just follow this setup. It shows how to set up public key authentication for the new user and disable password authentication.

Be extra careful when editing the server’s SSL configuration. It is scary simple to get locked out of a server.

Secure shared memory

Shared memory can be used in an attack against a running service, apache2 or httpd for example. More info here. To secure the shared memory:

sudo vim /etc/fstab

Add the following line:

tmpfs     /run/shm     tmpfs     defaults,noexec,nosuid     0     0

Save and reboot.

Disable SSL v3

Here’s why you want to disable SSL v3.

TLTR; it’s broken and vulnerable.

sudo vim /etc/apache2/mods-available/ssl.conf

Change the line:

SSLProtocol all -SSLv3

To:

SSLProtocol all -SSLv2 -SSLv3

Restart apache:

sudo apache2ctl configtest && sudo service apache2 restart

Setup fail2ban

Fail2ban temporarily rejects IP addresses that are trying to do something dodgy with your server. A classical example is a client try to brute force the SSH password. The default Fail2ban configuration only protectes SSH, but it’s (relatively) easy to configure it to add more checks (eg. apache HTTP server)

sudo apt-get install fail2ban -y

Installing applications

LAMP Stack

Before installing any application, we want to install Apache2, MySQL and PHP.

You can get away with a one liner:

sudo apt-get install apache2 mysql-server php libapache2-mod-php php-mcrypt php-mysql

But you really want to follow this thorough guide, from the guys @DigitalOcean .

Resilio Sync

Resilio Sync is the new incarnation of (the formerly known) BitTorrent Sync. In June, BitTorrent inc. announced that the Sync product was going to be moved into a new company, called Resilio. This news made me a bit nervous, but so far it seems that the new company is delivering. They recently released version 2.4 of the Sync client.

Resilio Sync is essentially a Dropbox replacement: you should be aware that it is a closed-source software. There are open-source alternatives, such as SyncThing or Seafile. I consider BitTorrent Sync (now Resilio Sync) the easiest to setup and to run and the more reliable.

Before installing the Sync client, create the folder where the synchronized files will be stored.

# change your-user with the directory name of the user created before
sudo apt-get install acl 
mkdir /home/your-user/btsync_share/

sudo setfacl -R -m "u:rslsync:rwx" /home/your-user/btsync_share/

To install Resilio Sync, start by creating a new apt source file

echo 'deb http://linux-packages.getsync.com/resilio-sync/deb resilio-sync non-free' | sudo tee --append /etc/apt/sources.list.d/resilio-sync.list

Add public key with the following command:

wget -qO - https://linux-packages.getsync.com/resilio-sync/key.asc | sudo apt-key add -

Install

sudo apt-get update && sudo apt-get install resilio-sync

Start Resilio Sync

sudo service resilio-sync start

The Linux version of Resilio Sync has a web-based UI available at 127.0.0.1:8888. The web UI is not accessible from the Internet, so we need to set up a reverse proxy.

First enable the Apache modules required for the reverse proxy to work:

sudo a2enmod proxy
sudo a2enmod proxy_http
sudo a2enmod rewrite
sudo a2enmod deflate
sudo a2enmod headers
sudo a2enmod proxy_balancer
sudo a2enmod proxy_connect
sudo a2enmod proxy_html

Create the Apache configuration for the Resilio Sync UI. Here, I assume that you have a domain name associated to your server and that each application will correspond to a sub-domain

Let’s pretend that the domain name is awesomehosted.com

Create a new Apache configuration file

sudo touch /etc/apache2/sites-available/btsync.awesomehosted.com.conf

Copy the following text to the new file

Enable the configuration and restart Apache

sudo a2ensite btsync.awesomehosted.com.conf
sudo service apache2 restart

If the sub domains are already configured and propagated, you should be able to access the Resilio Sync console

http://btsync.awesomehosted.com

In case you haven’t configured the A records yet, just add an entry to the /etc/hosts of the computer testing the connection

[server ip] btsync.awesomehosted.com

Baïkal

Baïkal is an excellent CalDav and CardDav server. I have been using it for a couple of years, and it totally replaced any other cloud service for syncing contacts and calendar (Apple Cloud, etc.). It is very stable and (kind of) easy to setup. The data are stored on SQLite, but for large dataset it is reccommended to use MySQL. I use SQLite because it’s easy to back up :)

# install the dependencies
sudo apt-get install php7.0-sqlite3 php-xml php7.0-mbstring unzip

# fetch baikal
cd /var/www
sudo wget https://github.com/fruux/Baikal/releases/download/0.4.6/baikal-0.4.6.zip
sudo unzip baikal-0.4.6.zip 

# rename the baikal folder 
sudo mv baikal dav.awesomehosted.com
sudo rm baikal-0.4.6.zip

# fix permissions
sudo chown -R www-data:www-data dav.awesomehosted.com

Create a new Apache configuration file

sudo touch /etc/apache2/sites-available/dav.awesomehosted.com.conf

Copy the following text to the new file

And enable the configuration

sudo a2ensite dav.awesomehosted.com.conf
sudo service apache2 restart

Hit dav.awesomehosted.com on your browser and you should access the configuration page. Choose an admin password and pick MySQL or SQLite. If you choose MySQL, you’ll have to create a database first. I don’t bother creating a MySQL db for Baikal. I have around 1000 contacts and a relatively low number of entries in my calendars and never had a problem. Bonus: SQLite is easier to backup!

As an Android user, I use CardDAV-Sync to syncronize my contacts and CalDav-Sync for the calendar syncronization.

Once you have created a new user (eg. johndoe), the url for the contacts is: http://dav.awesomehosted.com/card.php/addressbooks/johndoe/default/. The url for the calendar is: http://dav.awesomehosted.com/cal.php/calendars/johndoe/default/.

Tiny Tiny RSS

Tiny Tiny RSS (tt-rss for brevity) is a marvellous feed reader, which I use daily. There are lots of self-hosted feed readers out there: I have tried few of them, and I settled with tt-rss. The set-up process is similar to the one used for the other applications.

Install some required PHP dependencies.

sudo apt-get install php-curl php7.0-mbstring php-xml
sudo service apache2 restart

Clone tt-rss

cd /var/www

sudo git clone https://tt-rss.org/git/tt-rss.git tt-rss

sudo mv tt-rss rss.awesomehosted.com

sudo chown -R www-data:www-data rss.awesomehosted.com

Create a new Apache configuration file

sudo touch /etc/apache2/sites-available/rss.awesomehosted.com.conf

Copy the following text to the new file

And enable the configuration

sudo a2ensite rss.awesomehosted.com.conf
sudo service apache2 restart

Create the Tiny Tiny RSS database. Don’t forget to keep the password in a safe place.

mysql -u root -p

mysql> create database ttRssDB;
mysql> grant all on ttRssDB.* to ttRss@localhost identified by 'password';
mysql> flush privileges;
mysql> \q

The first time you hit the url ‘http://rss.awesomehosted.com’, a set-up page is displayed. Admittedly, the setup process of tt-rss it’s a bit quirky. Fill the configuration form with the information provided during the db creation stage. Click on the “Test Configuration” button to double-check you can access the database and all required libraries are present. Finally, select “Initialize Database”. Now you should be presented with a textbox containing the php configuration code required to run tt-rss. Press the “Install Configution” button. This will create a config.php file in the root of the main application directory.

You should now be able to access tt-rss. The default username and password is: admin/password. Don’t forget to create a new user and change the administrative password!

The last step required for a fully functional feed reader is creating a cronjob to refresh the feeds at a specified interval.

crontab -e

*/30 * * * \* /usr/bin/php /var/www/tt-rss/update.php --feeds --quiet

Docuwiki

My wiki is probably the most important part of my digital life. I keep track of books I read or I want to read, movies, a ’90s style weblog, ideas and notes. Before starting a personal wiki, I was an heavy user of Evernote. The problem with a note-style knowledge management system, such has Evernote, is that it quickly becomes a huge knowledge sink and the truly important bits of information that I want to retain were lost in the noise of the less important data.

A wiki instead forces me to curate the information that I want to preserve. A wiki is akin to having a paper notebook. It is slower, but it forces me to think about the actual bit of data that I’m working on.

Docuwiki is a very simple wiki platform with a couple of advantages over other wikis: it has a large plugin ecosystem and it is file-based, so super-easy to back-up. No database, less attack-surface. The installation is also straightforward.

cd ~

# download the latest version of docuwiki
wget http://download.dokuwiki.org/src/dokuwiki/dokuwiki-stable.tgz
tar xvf dokuwiki-stable.tgz
sudo mv dokuwiki-2016-06-26a/ /var/www/wiki

# setup folder permissions
sudo chown www-data:www-data -R /var/www/wiki/

# create apache conf file
touch /etc/apache2/sites-available/wiki.awesomehosted.com.conf

Copy the following configuration snippet to the new file

Restart Apache and test the installation by browsing to http://wiki.awesomehosted.com.

Piwigo

Piwigo is a photo gallery web-application. I use it to share photo galleries with my family. I switched from the very stylish-looking Lychee to Piwigo because Lychee doesn’t support custom ordering of photos in a gallery.

Piwigo setup is straightforward. Start by creating a database. Don’t forget to change the password.

mysql -u root -p

CREATE DATABASE piwigo;

CREATE USER piwigouser@localhost IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON piwigo.* TO piwigouser@localhost;

FLUSH PRIVILEGES;

exit

cd ~

sudo apt-get install unzip php-gd

wget "http://piwigo.org/download/dlcounter.php?code=latest" -O piwigo-gallery.zip

sudo unzip piwigo-gallery.zip -d /var/www

sudo mv /var/www/piwigo /var/www/gallery.awesomehosted.nl

sudo chown www-data: -R /var/www/gallery.cryptonomicon.nl/

Create the Apache config file

sudo touch /etc/apache2/sites-available/gallery.awesomehosted.com.conf

Copy the following configuration to gallery.awesomehosted.com.conf

Finally, add the following lines to my.cnf

sudo vim /etc/mysql/my.cnf

[mysqld]
sql-mode=""

This is required to avoid a weird error when changing the order of the photos in the gallery.

Secure the applications with TLS/SSL

Once all the applications are installed, configured and running on port 80 within Apache 2, it’s time to install a TLS/SSL certificate in order to encrypt the traffic between client and server. The obvious choice these days is to use Let’s Encrypt

Let’s Encrypt is a new SSL authority that provides free SSL certificates and a super-simple command line tool to streamline the creation and renewal of certificates.

Let’s start by installing the command line tool.

sudo apt-get install python-letsencrypt-apache

Modify the Apache 2 default virtual host file, 000-default.conf so that the domain name of your server is properly defined.

sudo vim /etc/apache2/sites-available/000-default.conf

Make sure that the ServerName and ServerAlias are pointing to the correct domain name - let’s continue using the fictional awesomehosted.com domain name:

ServerName awesomehosted.com
ServerAlias www.awesomehosted.com
ServerAdmin webmaster@localhost
DocumentRoot /var/www/awesomehosted

I like to configure the root directory with my own index.html, hence DocumentRoot pointing to a non default folder.

Finally, execute the command that will magically install and configure the SSL certificate for the main domain and the subdomains.

letsencrypt --apache -d awesomehosted.com -d www.awesomehosted.com -d btsync.awesomehosted.com -d dav.awesomehosted.com -d rss.awesomehosted.com -d wiki.awesomehosted.com -d gallery.awesomehosted.com

Note that his command will fail if the DNS settings for the domains and subdomains are not properly configured and propagated.

Stay tuned for the second part of these notes, concerning my backup strategy.

Mutt 1.7.0 on El Capitan

2016-08-21T00:00:00+00:00

This is an update to my one year old post about mutt and OS X. On August 2016, a new version of mutt was released. This version includes the sidebar patch and the trash folder patch, so no need to mess around with patches any longer. Here are my notes to get started with the new version.

If you have previously installed mutt via Homebrew, remove it.

brew uninstall mutt

Download mutt and untar the archive.

wget ftp://ftp.mutt.org/pub/mutt/mutt-1.7.0.tar.gz
tar xopf mutt-1.7.0.tar.gz

Configure and build.

export CPPFLAGS=-I/usr/local/opt/openssl/include
./configure --enable-imap --enable-smtp --enable-sidebar --enable-gpgme --with-regex  --enable-locales-fix --with-ssl --with-sasl --with-gss --enable-hcache --with-tokyocabinet

make

make install

I had to link to my (brew based) openssl installation folder for the --with-ssl flag to work. You also need to install Tokyo Cabinet for the header cache flag to work (brew install tokyo-cabinet).

That should be it. Do not forget to set the trash option in your .muttrc file. Mine is configured like so:

set trash=imaps://$my_server/INBOX.Trash

For additional information on how to configure mutt on OS X, take a look at my previous post.

Getting started with Mutt on OSX

2015-05-29T00:00:00+00:00

20/08/2016 I wrote a short upgrade to this guide, that uses the newly released mutt 1.7.0.

Here are some notes on how to get started with mutt and Fastmail on OSX (Yosemite). Most of this stuff applies to Linux as well. You can actually skip this whole mutt-is-awesome rant and checkout the “mutt” section of my doftiles.

Why mutt

mutt is stable, solid, predictable and flexible mail client. Also, I like writing in Vim. And I spend a lot of time in the command line. In the past, mutt was a pain to setup. Just google for “mutt setup” and you will find some monster articles, like this one or this one. Scary shit, uh?

A “normal” setup would include mutt, offlineimap, some smtp client, like msmtp and notmuch for searching your emails. Enough to scare away even the most determined hacker.

My setup is way simpler, since mutt is able to send emails without a third-party smtp client (I don’t know exactly when that option has been introduced in mutt). Additionally, I don’t require to store my emails locally, I don’t need to reply when off-line and I can search using the excellent Fastmail web interface.

This is how mutt looks like:

Get mutt

Vanilla mutt lacks some of the essential stuff, which is only available through patches. One is the trash patch - as in, send message to trash - and a sidebar to show IMAP folders. Applying a patch is not complicated - get the source, get the patch, make. In OSX, I can simply use Homebrew and get it done with a simple command.

brew install kevwil/patches/mutt \
    --with-gpgme --with-trash-patch \
    --with-sidebar-patch --with-confirm-attachment-patch

Talking to fastmail

mutt is configured through a .muttrc file. It’s a good idea to keep your configuration well organized and commented, because it can get a messy.

This is the very basic configuration required to fetch and send email through Fastmail.

set my_server       =   mail.messagingengine.com
set my_smtp_server  =   mail.messagingengine.com
set my_user         =   xxx@fastmail.fm
set my_pass         =   "`security find-internet-password -g -a lfi@fastmail.fm 2>&1| perl -e 'if (<STDIN> =~ m/password: "(.*)"$/ ) { print $1; }'`"
    
set record          =   "imaps://$my_server/INBOX.Sent Items"
set postponed       =   "imaps://$my_server/INBOX.Drafts"
    
set from            =   "xxx@fastmail.fm"

# Account - SMTP

set smtp_url        = "smtp://$my_user:$my_pass@$my_smtp_server:587"
set smtp_pass       = $my_pass
set imap_user       = $my_user
set imap_pass       = $my_pass
set ssl_force_tls   = yes
set ssl_starttls    = no
    
#
# Default inbox
#
set spoolfile=imaps://$my_server/INBOX

#
# Default location of mailboxes
#
set folder=imaps://$my_server/INBOX

This is good enough to start reading and sending emails with Fastmail. If you use gmail, there are plenty of tutorials out there that shows an equivalent configuration. You may have noticed that the my_pass variable is using the security command to retrieve the password from the OSX Keychain. I figured it was a good idea to remove my password from the .muttrc file, since I store all this stuff publicly on github.

To add the password to the OSX Keychain, type:

sudo /usr/bin/security -v add-internet-password -a xxx@fastmail.fm -s mail.messagingengine.com -w your-password

Contacts

mutt uses aliases to map a name to an email address. Aliases are stored in a file using a simple format:

alias john John Doe <johndoe@email.com>

The default key binding to add an address from the currently opened mail is a.

I store my contacts in a OwnCloud instance running on my server. Initially, I used a script to convert the vCard contacts into an alias flie, but it has been a sub-optimal solution, because I had to download the vCard file and re-import periodically.

Enter pycarddav. pycarddav is a CardDav CLI client with built in support for mutt’s query_command.

The setup takes a couple of minutes:

Install pycarddav:

pip install pycarddav

Create a configuration file with the credentials of the CardDav server and syncronize:

pycardsyncer

Test:

pc_query Barack
...
Name: Barack Obama
TEL (CELL, VOICE, pref): 123446767
TEL (WORK, VOICE): 999999999
EMAIL (INTERNET, HOME, pref): barack@whitehouse.org

Now, it is possible to configure mutt to search the contacts database using TAB when composing. Add this entry to the mutt .muttrc file.

set query_command="pc_query -m '%s'"

The sidebar it’s an handy right-side panel that shows a configurable number of IMAP folders and the number of messages that each folder contains. I keep it hidden by default, and display it when needed.

The folders displayed in the sidebar must be specified in the .muttrc file, like so:

mailboxes "+-----------------" \
      imaps://$my_server/INBOX \
      "imaps://$my_server/INBOX.Sent Items" \
      "imaps://$my_server/INBOX.Drafts" \
      "+-- folders ------------" \
      "imaps://$my_server/INBOX.folder1" \
      imaps://$my_server/INBOX.folder2 \
      imaps://$my_server/INBOX.folder3.subfolder \
      "imaps://$my_server/INBOX.folder4" \
      "imaps://$my_server/INBOX.folder5" \
      "imaps://$my_server/INBOX.folder6" \
      "imaps://$my_server/INBOX.folder7" \
      "imaps://$my_server/INBOX.folder8" \ 

Fine-tuning

It’s easy to geek out with mutt’s configuration. There are tons on parameters. My approach is to start with a simple configuration and snoop at other’s people .muttrc files. These are some settings that I have been using since start using mutt.

set sort              = threads                     
set sort_aux          = reverse-last-date-received 

Gmail-style threading.

set pager_stop

Stop mutt showing the next message when we are at the end of the current message’s body.

set fast_reply
set include     

Quick reply, makes mutt to stop asking lots of questions.

set reverse_name 
set envelope_from

Pre-fills the “From” address when replying to emails, based on the email account that received the original mail. Useful in case of multiple accounts or multiple aliases (like in my case).

As I mentioned early, I like writing in Vim. This is how I tell mutt to open Vim when composing an email:

set editor = "vim +/^$ ++1 -c 'set tw=76 expandtab nosmartindent spell'""

Here are a bunch of mutt dotfiles that I have used for inspiration.

Bao Nguyen dot files

Jeff Waugh mutt configuration

Dave Pearson mutt configuration

Angelo Olivera dot files

Hong Shick Pak dot files

GPG

mutt and GPG works pretty much hand in hand, no biggie, given that you have your GPG stuff all setup (I assume that if you made it so far, you are probably an avid GPG user). My configuration is shamelessly copied from this 2014 article on mutt and GPG. Just remember to brew mutt with the --with-gpgme flag and install gpgme (brew install gpgme).

I like to sign all my outgoing mail. Easily done with this option set.

set crypt_autosign=yes

With recent version of GPG, mutt output an error on startup that says Using GPGME backend, although no gpg-agent is running. I think there is a patch floating around, but I didn’t bother applying it, because it works.

Markdown

I use Roguelazer’s muttdown to send HTML email written in markdown. Before muttdown, which is a quite recent project, I was using a mutt macro based on pandoc (something similar to what is described here). muttdown is a sendmail replacement that transparently compiles annotated text/plain mail into text/html.

I just need to start my message with !m to trigger the markdown compilation.

To use muttdown:

git clone https://github.com/Roguelazer/muttdown.git
cd muttdown
python setup.py install

Create a configuration file, .muttdown.yaml

smtp_host: mail.messagingengine.com
smtp_port: 587
smtp_ssl: false
smtp_username: xxx@fastmail.fm
smtp_password_command: security find-internet-password -w -a xxx@fastmail.fm        

In .muttrc add:

set sendmail = "muttdown -c /path/to/.muttdown.yaml"

Bindings and macros

The default key binding for mutt is quite reasonable. mutt allows to bind the same keystroke to different action in each view (index, pager, browser, attach, etc. - mutt calls the views “menus”).

I favor a Vim-like binding, mostly for navigating the email list and the email body.

mutt offers rudimentary macros as well. Macros in mutt are essentially a way to automate keystrokes.

To get an idea of how a macro looks like, take a look at this macro. It is triggered by the TAB key and it’s only available in the index menu (the message list). When triggered, it shows the next unread message and press enter.

macro index   <tab>      <next-unread><enter>

Here is my binding file.

This is an awesome, interactive mutt cheatsheet for getting started with the key binding.

HTML emails

You can throw any kind of attachment to mutt, as long as the type is specified in a mailcap file. My mailcap file looks like this:

application/msword; ~/.mutt/view_attachment.sh %s "-" '/Applications/TextEdit.app'
#
# # Images
image/jpg; ~/.mutt/view_attachment.sh %s jpg
image/jpeg; ~/.mutt/view_attachment.sh %s jpg
image/pjpeg; ~/.mutt/view_attachment.sh %s jpg
image/png; ~/.mutt/view_attachment.sh %s png
image/gif; ~/.mutt/view_attachment.sh %s gif
#
# # PDFs
application/pdf; ~/.mutt/view_attachment.sh %s pdf

#
# # HTML
text/html; ~/.mutt/view_attachment.sh %s html; copiousoutput
#
# # Unidentified files
application/octet-stream; ~/.mutt/view_attachment.sh %s "-"

The “mailcap” file is just a way to map a MIME type to an application that can handle it. I’m using the view_attachment.sh script, which is mentioned in the mother of all mutt articles. It works well for me.

Once the “mailcap” file is sorted, mutt can auto-open HTML emails with an entry in the .muttrc file:

  auto_view text/html                                      # view html automatically
  alternative_order text/plain text/enriched text/html     # save html for last

Colors

I’m lazy, so I use the Solarized theme for mutt, available here.

1Password on Linux

2015-02-22T00:00:00+00:00

Lately, my long-lasting love for Apple products has been fading. I ditched my iPhone 6+ for a OnePlus One and I relegated my MBP to mostly photo-editing tasks. I’m typing this from a fresh installation of Xubuntu 14.10, which I have been using in the past and I tend to prefer to the heavy-weight Ubuntu.

My totally most used app in the Apple eco-system is Agile Bits 1Password. In fact, what I miss most in my new Android phone is the fingerprint reader integration with 1Password. I have to type my rather long and cumbersome password every time I need to look for a piece of encrypted info. Agile Bits doesn’t support Linux at all. For Linux users, they offer a web interface called 1Password Anywhere. It’s Javascript based and runs locally. Sub-optimal, if you ask me.

So, during the Xubuntu setup process, I started googling around for some recent solutions to running 1Password on Linux and I stumbled on the awesome, little repo of Ryan Gordon. Ryan has hacked together a Lua script, named 1pass, which can decrypt the 1Password keychain and show whatever info you store in 1Password in a very minimalistic GUI.

The setup wasn’t exactly straightforward, but eventually I got it to work. Here is quick rundown of the steps required to get going with 1pass.

InstaIl Mercurial, to fetch the 1pass repo.

sudo apt-get install mercurial

Clone Ryan’s repo, 1pass:

hg clone http://hg.icculus.org/icculus/1pass/

Install some dependencies:

sudo apt-get install libgtk2.0-dev libxtst-dev

Run the build:

cd 1pass
./build.sh

If 1pass built without errors, copy (or better symlink from Dropbox) the keychain in 1pass folder.

mkdir 1Password
ln -s ~/Dropbox/xx/yy/zz/1Password.agilekeychain/ 1Password/1Password.agilekeychain

You can finally try out 1pass, by typing:

./1pass

The script welcomes the user with a rather cryptic message:

Now waiting for the magic key combo (probably Alt-Meta-\) ...

The “magic combo, on my Lenovo X220, is “Windows Key+Alt+" I had to figure it out by checking out a keyhook.c file, located in the project’s root. For the binding to work, you need to press the keys in the right sequence, which is:

Alt -> Win -> \

By pressing the keys at the same time, or the Win key first, will cause the popup to stay well hidden.

This is the password prompt box that pops-up on pressing the “magic combo”.

On successful password entry, a floating menu displaying the categories in the keychain should appear. You can then navigate through the categories, select the password, press Enter and the password will be copied in the clipboard. The key binding works everywhere, so it is possible to access the encrypted info’s menu from the browser or whatever application.

To avoid leaving a terminal tab opened just for 1pass, I go like this:

./1pass & exit

Edit:

This post was picked up by Hacker News. As usual, lots of interesting pointers. Follow the conversation here: https://news.ycombinator.com/item?id=9091691

Vim configuration for happy Java coding

2013-10-04T00:00:00+00:00

During the last couple of years, I gradually moved to Vim for editing text and writing code. It’s not an easy journey: Vim requires dedication and it’s sheer amount of options and plugins can be intimidating for a newcomer. However, with enough perseverance, the VIM’s approach to manipulating text becomes natural and you will not look back (enough with the Vim advocating, there are countless blog posts discussing the marvels of Vim).

In this post, I’m presenting my current Vim configuration for working with Java and Groovy projects. There are lots of folks who run away screaming when you try to suggest they abandon one of the mainstream IDEs (Eclipse, IntelliJ, Netbeans) and move to Vim. How will you do without auto-completion? How do you navigate files? How do you refactor? How do you jump to a function definition? It turns out that you can do most of that stuff in VIM. Refactoring it’s probably the only functionality that is hard to achieve, even if it’s not an impossible goal (Eclim partially supports that).

I’m not going to try to convince you that Vim is better than your favorite Java IDE, especially if you have never used Vim before. I have never felt very comfortable working inside one of the heavyweight IDEs and my productivity is just higher when I work with my VIM setup. Your mileage may vary, but I encourage you to take a spin.

(As a side note, James Gosling famously stated that you’d better use an IDE if you want to code in Java.)

This setup has been tested on OSX 10.8 and MacVim.

Setup

I assume you use Pathogen for managing your VIM plugins. If you don’t, I highly recommend you do (or look into other bundle managers, like Vundle).

If you are on a Mac you should use MacVim and you should install it using Homebrew.

Java and Groovy syntax highlighting

MacVim comes with a truckload of syntax highlighting files, including Java and Groovy.

In order for syntax highlighting to work, add

syntax on

to your .vimrc file.

This is how Java and Groovy files look like in Vim:

If you are using a recent version of MacVim (7.4 or higher) chances are that the Groovy syntax file is broken.

In order to fix the syntax issue, open the groovy.vim file located in /usr/local/Cellar/macvim/7.4-xx/MacVim.app/Contents/Resources/vim/runtime/syntax and comment out line 260.

Snippets

Snippets are essential to decrease typing and increase the speed of development. All modern IDEs support snippets (also known as templates) with different level of configurability.

After testing different snippet plugins for Vim, I settled with UltiSnips. It requires a version of Vim compiled with Python. If you use MacVim installed with Homebrew, you should be all set.

To check if your Vim installation is compiled with Python support, type:

$ vim --version | grep python
+cryptv   +linebreak    +python    +viminfo
+cscope   +lispindent   -python3   +vreplace

If you see +python you are good to go.

Install UltiSnips in your ~/.vim/bundle folder:

git clone https://github.com/SirVer/ultisnips.git

Ultisnips has several Java snippets already configured and ready to use. You can create new snippets by adding a java.snippets file to a ~/.vim/snippets folder.

If you add your own snippets, do not forget to configure Ultisnips to access also your own snippets.

let g:UltiSnipsSnippetDirectories=["UltiSnips", "snippets"]

This is an example of UltiSnips in action:

UltiSnips snippets use placeholders for inserting custom data:

snippet co "constant" b
public static final ${1:String} ${2:var} = $3;$0
endsnippet

The previous snippet (triggered by co) create the following statement:

public static final String var = ;

and places the cursor on String. By pressing <CTRL+j> you can move the cursor to the remaining placeholders, defined with ${1}, ${2} and so on. The placeholder marked with ${0} is always the last one the cursor will move to.

Indent highlight

Java and Groovy code can display several levels of indentation. To help keeping track of the code blocks I use the IndentLine plugin. There are several other plugins that offer the same functionality, but I kind of like the simplicity of this one.

To install IndentLine, clone the repo into your ~/.vim/bundle folder:

git clone https://github.com/Yggdroot/indentLine.git

I configure mine like so:

" vertical line indentation
let g:indentLine_color_term = 239
let g:indentLine_color_gui = '#09AA08'
let g:indentLine_char = '│'

Please note that Vim natively supports visual guides for tab-indented blocks. This StackOverflow answer clearly shows the settings to enable indentation guide and remove tabs before saving a file.

Auto completion

Vim basic auto completion for Java is an essential feature for working with third-part libraries and large code bases. On my quest for the perfect Java/Vim setup, I started using Javacomplete which never really worked as well as any IDE in terms of auto completion. I knew of the existence of Eclim but I kind feel like cheating to resort to an headless Eclipse for my auto completion needs. Additionally, I considered the setup a bit too complex. Eventually, I decided to try out Eclim and have been very satisfied with the Vim integration. Eclim documentation is excellent and the bundle integrates well with other plugins, like supertab (more on supertab later).

The only drawback is that you need to install Eclipse. I consider IntelliJ a superior platform but I still use Eclipse with the Geppetto plugin to edit Puppet files.

Installing Eclim

The installation page for Eclim is quite comprehensive. The basic installation steps are:

wget http://downloads.sourceforge.net/project/eclim/eclim/2.3.2/eclim_2.3.2.jar
java -jar eclim_2.3.2.jar

eclim 2.3.2 is the package to download if you have Eclipse 4.3.x (Kepler). There is a different package for older versions of Eclipse.

The installer has a graphical interface and it only requires to select the current Eclipse installation and additional packages (for example, Scala). It takes just a couple of minutes.

Once the installation is completed, you want to start the Eclipse headless daemon using:

$ECLIPSE_HOME/eclimd

The boot procedure takes around 30 seconds.

To test that Vim can “talk” to Eclipse, open Vim and type

:PingEclim

If your output says something like

eclim   2.3.2
eclipse 4.3.0

you should have a working integration.

Finally, you must create a new Eclipse project or import an existing one.

My workflow revolves around Gradle, a Groovy based build tool which is getting a phenomenal amount of traction. This is what I do (using a script):

mkdir my-awesome-project && cd $_
gradle setupBuild --type java-library
perl -i -lne 'print $_;print "apply plugin: \x27eclipse\x27" if(/apply/);' build.gradle    
gradle eclipse

Then from inside Vim:

:ProjectImport .

The first line creates a new folder named my-awesome-project. The second line calls Gradle’s project generation feature (similar to Maven archetypes). The third line adds the Eclipse plugin to the Gradle build file. Finally I invoke gradle plugin to generate the Eclipse project files.

In this short video, you can see a couple of Eclim’s features:

Automatically adding an import (:JavaImport)
Automatically adding a getter and setter (:JavaGetSet)

For a complete list of Eclim commands, please refer to this cheatsheet.

Supertab

The second auto completition plugin is Supertab. It essentially allows completing using the <TAB> key instead of the standard <CTRL+X><CTRL+N>, when in insert mode. It plays nicely with Eclim, even though some fiddling is required.

To install Supertab, clone the repo in your .vim/bundle folder:

git clone https://github.com/ervandew/supertab.git

Delimitmate

DelimitMate provides automatic closing of quotes, parenthesis, brackets, etc. A real time saver. The plugin works pretty well out-of-the-box and I didn’t find any compatibility issue with the other plugins.

To install Delimitmate, clone the repo in your Pathogen bundle folder.

git clone https://github.com/Raimondi/delimitMate.git

This is the only setting I have modified for this plugin:

let delimitMate_expand_cr = 1

Check out help delimitMate to learn about other options that may better serve your workflow.

File browsing

Eclim can display a project navigation pane by issuing the :ProjectTree command, but I still prefer NERD tree to visualize my project’s files.

NERD tree is a very popular plugin with tons of documentation: these blog posts contain all the information to get you started with this plugin.

Install NERD tree by cloning the Github repo in your bundle folder.

git clone https://github.com/scrooloose/nerdtree

My NERD tree configuration is kept to a minimum: I have only mapped <CTRL-d> to toggle the file browsing pane.

nmap <leader>d :NERDTreeToggle<CR>

CtrlP

If you have been using Vim for a while, CtrlP is surely in your bundle folder. It’s a fuzzy finder for files, buffers, MRU and even tags. I have mapped CtrlP to <CTRL+t> and I can use the binding to type the first letters of the file I want to open. CtrlP is the equivalent of <CTRL + ALT + N> in IntelliJ Idea and <CTRL-SHIFT-T> in Eclipse.

For example, if I want to open a file named LibraryTest.java I can type <CTRL+t LT> (LT = LibraryTest). This plugin is indispensable when working on large projects with hundreds of files.

Install CtrlP by cloning the repo in your Pathogen’s bundle folder:

git clone https://github.com/kien/ctrlp.vim

These are my setting for the plugin:

" Use <leader>t to open ctrlp
let g:ctrlp_map = '<leader>t'
" Ignore these directories
set wildignore+=*/build/**
" disable caching
let g:ctrlp_use_caching=0

Building

The last part of my Vim setup is related to building and running Java or Groovy tests. As I have mentioned previously, I use Gradle to build my Java/Groovy projects.

Gradle works similarly to Ant or Maven. A build.gradle file is placed in the project’s root directory and a Gradle process runs the build based on the build file content.

A Gradle build is launched from the command line like so:

gradle clean build
gradle test

Vim is built with support for compiler integration and the MacVim distribution comes with over 60 compiler configurations. Unfortunately Gradle is not among the supported compilers so I had to create my own Gradle compiler file, with a little help from Niklas.

With the Gradle compiler file in place, I can now set the compiler to Gradle using :compiler gradle and run my tests by typing make test. This solution works for both Java and Groovy projects, as long as you use Gradle as a build tool. I have mapped everything to the <F4> key, like so:

map <F4> :w<CR> :compiler gradle<CR> :make test<CR>:cw 4<CR>

When pressing <F4>, Vim saves the current file, run the Gradle compilation and open the QuickFix window if there are compilation errors or test do not pass.

There is one gotcha: the QuickFix window is supposed to show compilation errors that you can “navigate” using <:cn> or <:cp> and even jump to the file reported in the errors list. This functionality is based on the assumption that the compiler output is properly parsed, which is not the case for Gradle. Still, I’m able to display the compilation/test errors in the QuickFix window.

I’m currently working on a solution (based on this forum entry) for having Gradle and the QuickFix window to play together nicely.

This is how the test feedback loop looks like:

Maven

In case you Maven is your build tool of choice, you can use Eclim to build the project using the Eclim command :Mvn.

vim-rooter

One more plugin is required for the compilation to work correctly in my setup. By default, the :make command is executed inside the directory of the currently open file. If I have a test case opened in src/test/java/my/awesome/package and I call :make, the Gradle process will be executed inside that directory and will obviously fail. Gradle requires a build.gradle in order to work. We need Vim to change the current directory to the project’s root folder. The vim-rooter plugin has this exact functionality. It detects the project root by the presence of known directories, like .git or files like Rakefile.

It’s a simple plugin that doesn’t require any additional configuration. To install it, clone the Github repo into your bundle folder.

git clone https://github.com/airblade/vim-rooter.git

I hope that this rather lengthy guide will encourage you to try out Vim for your daily development activities.