Server Ubuntu 12.04

Target is to setup an Ubuntu Home server with Dynamic DNS access.

The actual Ubuntu version is 12.04 LTE (April 2012, Long Term Edition, updates for Desktop 3 years, security updates for 5 years).

First I tried to use a RAID 1 disk array. Because it did not work reliable, I use now a 16 GB SSD.

Services: http://www.moinmo.in Wiki, Typo3 CMS, Proxy for a webcam computer.

Following most of the special software installation is described.

Power Supply: 230 VAC, 11 - 12 W, with battery backup.

RAID 1 support

2012-09-13 After a hardware defect of the Asus 900A netbook, also the RAID1 array disk were corrupted. I no longer see a good reason to use a RAID1 array. It seems to be better to use a reliable SSD-disk and a backup drive (SD-card or harddisk). The program Back in Time was selected for that purpose, see below.

Last time with Ubuntu 10.04 I worked about 3 month to get RAID1 to run, because I did everything manual. The Desktop edition of Ubuntu does not have RAID support included.

The Ubuntu Server edition has RAID support, but no GUI. I use the GUI mostly with VNC (Remote Desktop).

Then I found out by accident, that the Ubuntu Alternate edition has RAID support and GUI. So, this time with Ubuntu 12.04 I gave it a try. OK, the installation of RAID1 went good, but the GRUB installation not. Then I tried the good program Boot-Repair, but unfortunately it did not work. The last try with chroot from the Alternate installation worked. From the BIOS I can boot either hard disk sdb, sdc and RAID1 works.

The package mdadm installs also postfix, and you can set it up to send any degration error to your mail address, as good NAS (Network Attached Storage) do. That works in my case. A year ago I had to replace an USB hard disk, without data loss.

Two years ago a netbook was the far cheaper solution for a RAID1 Home Server, regarding computing power, low power consumption and reliability. A netbook even has a power fail system for a short interrupt (battery).

The computer is an Asus netbook P900 (Atom CPU, 1 GB RAM, no CD-ROM).

Mass Storage are two USB 2.0 harddisks (each about 250 GB) in a RAID1 array.

Ubuntu 12.04 Alternate USB-Stick

The Alternate Edition of Ubuntu is for installation only, no live version.

The download page for Ubuntu 12.04 LTS is here.

The standartd i386 alternate version is selected.

Use http://unetbootin.sourceforge.net/ to bring the ISO version on to an bootable USB-stick, which is available for Linux, Mac OS X and Windows. It will allow also a persistent area on the USB-stick (>= 1 GB).

How to bring the iso version on to an bootable USB-stick under Mac OS X does not give a persistent area, it is just shown for demonstration.

# Job done under Mac OS X 10.7.4 Lion, Terminal

# change directory
$ cd Downloads

# convert ISO to IMG file
$ hdiutil convert -format UDRW -o ubuntu-12.04-alternate-i386.img ubuntu-12.04-alternate-i386.iso
Master Boot Record (MBR : 0) lesen …
Ubuntu 12.04 LTS i386            (Apple_ISO : 1) lesen …
 (Windows_NTFS_Hidden : 2) lesen …
...............................................................................................
Dauer:  7.592s
Geschwindigkeit: 90.5M Byte/s
Ersparnis: 0.0 %
created: /Users/rudi/Downloads/ubuntu-12.04-alternate-i386.img.dmg

# rename
$ mv ubuntu-12.04-alternate-i386.img.dmg ubuntu-12.04-alternate-i386.img

# show disk names
$ diskutil list

# figure out disk name, Type: DOS_FAT_32
  -> /dev/disk3

# umount USB-Stick
$ diskutil umountDisk /dev/disk3
Unmount of all volumes on disk3 was successful

# copy Ubuntu ISO to USB-Stick
$ sudo dd if=ubuntu-12.04-alternate-i386.img of=/dev/disk3 bs=1m
Password:
687+1 records in
687+1 records out
720678912 bytes transferred in 402.027790 secs (1792610 bytes/sec)

That works, but it is better to have a persistant version, were you can add programs.

Help for a RAID installation in general.

Help for a Software-RAID installation (2012-03).

Help for a Hotplug RAID installation (10.04).

Setup Ubuntu 12.04 Alternate

Provided hardware:

Software setup:

Setup hostname and IP

In order to use SSL encryption it is important to use the right hostname. Also the TCP/IP number should be the same as before the old server had.

# file /etc/hostname
rudiswiki.de

# file /etc/hosts
127.0.0.1       rudiswiki.de    
127.0.1.1       localhost.localdomain   localhost
...

# TCP/IP number: 192.168.17.72
# Setup in GUI network manager

Program Installation

A few programs should be installed:

Setup Vino (VNC server)

The VNC server is protected with a password (o..).

In Unity 3D clearing the screen on the client side does not work, because of the screen hardware acceleration, see VNC session. To change back to Unity 2D do the following:

At the beginning because of my German keyboard layout the letters y z are exchanged. It looks like, than after a reboot the keyboard has QUERTZ layout.

Setup postfix (email)

The setup should be done as described in UbuntuRaid1#Setup_Postfix_Email_send .

If everything ist OK test RAID monitor system:

$ sudo mdadm --monitor --test /dev/md0

# you should receive an email
Header: TestMessage event on /dev/md0:900A73
Body:
This is an automatically generated mail message from mdadm
running on 900A73

A TestMessage event had been detected on md device /dev/md0.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sdc1[1] sdb1[0]
      243156856 blocks super 1.2 [2/2] [UU]
      
unused devices: <none>

If another package has installed MTA exim it should be exchanged with postfix.

# Before you maybe have to change permissions:
$ sudo chown root:rudi aliases
$ sudo chmod g+w aliases

$ sudo apt-get install postfix

Setup apache2 web server

Apache2 needs some modification in the setup:

# enable modules
$ sudo a2enmod dav
$ sudo a2enmod headers
$ sudo a2enmod proxy
$ sudo a2enmod proxy_http
$ sudo apt-get install libapache2-mod-proxy-html
or
$ sudo a2enmod proxy_html
# Problem: proxy_html.load: Cannot load /usr/lib/libxml2.so.2 into server
# https://bugs.launchpad.net/ubuntu/+source/mod-proxy-html/+bug/964397
$ sudo ln -s i386-linux-gnu/libxml2.so.2 libxml2.so.2

$ sudo a2enmod ssl
# module wsgi for Python support
$ sudo apt-get install libapache2-mod-wsgi
or
$ sudo a2enmod wsgi

$ sudo service apache2 restart

In order to connect via network to the Ubuntu server the following connection can be setup in Finder:

# Network connection with SMB, with R/W, with the share option in Linux
smb://192.168.17.72

# Network connection with VNC Virtual Network Computing, Remote Desktop
vnc://192.168.17.72

# WebDAV, Web-based Distributed Authoring and Versioning
https://192.168.17.72/dav

# Network connection with AFP Apple File Protocol, with R/W
afp://192.168.17.72
# CAUTION: This will generate a lot of hidden (.) files.

For webDAV setup please see at AndroidNotes#Setup_webDAV_Storage.

Setup moin wiki

Because the moin installation is in the /home folder it can be copied from old server to new server. Just the permissions have to be adjusted.

# adjust permissions
$ sudo chown -R www-data:www-data moin-1.9.4
$ cd moin-1.9.4
$ sudo chmod -R ug+rwx wiki

# clear the cache
$ ./moin maint cleancache

Setup SSH

In order to work, the files /etc/hostname and the /etc/hosts have to setup properly, see ServerUbuntu1204#Setup_hostname_and_IP. If you change the server, even with the same host name and IP address, the SSH key is changed, so you have to renew all connections.

For Mac OS X the file /User/rudi/.ssh/known_hosts has to be changed. The target IP address 192.168.17.72 line must be deleted. At the next connection try it is asked if the new host is trustworthy. Answer with yes and the connection is stored again in the known_hosts file.

Setup Zarafa (Exchange server)

# create document root for apache2
$ sudo mkdir /var/www/sabre-zarafa

Data backup with Back in Time

The Ubuntu standard program deja dup (GUI for duplicity) I do not like, because it uses a proprietary archive format.

Because I made good experience with the Mac OS X program Time machine I looked for something similar, and found '''Back In Time''' (based on rsynch). There I could save the folders /etc, /home and /var every hour to the SD-card.

After the first full backup (snapshot), the following backups saves only new and changed files, in case of no data change only hard links are set in order to save space.

If the free space left on the backup media is less than 1 GB (parameter) old snapshots can be smart removed (Settings/Auto-remove):

Copy the Installation to another media

In order to make a bootable backup, or copy the (bootable) installation to another media, the following procedure proved to be working:

# Boot with an Ubuntu 12.04.1 live media, USB-stick with persistant area (dev/sdc1).
# USB-stick is prepared with program unetbootin (Linux, Mac OS, Windows, see link)
# Source media is an USB harddisk on /dev/sdb1 (file system ext4)
# Target media is an internal SSD on /dev/sda1 (file system ext4)

# Install program synaptic, in order to get the "univers" repository.
# Install the program ddrescue.
$ sudo apt-get install gddrescue

# The target media should have a partition size with similar size of the source partition.
# Partition resize is made with program Gparted.
# Example (ddrescue shows the progress and data rate):
$ sudo ddrescue -f /dev/sdb1 /dev/sda1 ddrescue.log
# ddrescue will report at the end of copying an error of partition size mismatch.

# Install the Grub boot loader:
$ sudo mount /dev/sda1 /mnt
$ sudo grub-install --root-directory=/mnt /dev/sda

# Fix partition size mismatch
$ sudo resize2fs /dev/sda1

# Check target media file system, must be OK to proceed
$ sudo fsck.ext4 /dev/sda1

# Shut down (remove all other media) and reboot with /dev/sda
# blood preasure will rise :-)
$ sudo reboot

The last action is to adjust /etc/fstab:

# Find UUID's of the drives
$ sudo blkid
/dev/sda1: UUID="89cb8d0c-200e-4022-8746-18603304d2c8" TYPE="ext4" 
/dev/sda2: UUID="fd0b068b-ca58-4bef-8772-468f6c21c441" TYPE="swap"  
/dev/sdb1: LABEL="SAVE" UUID="5924b296-d3eb-4323-bec6-da750b2642e8" TYPE="ext4"

# change /etc/fstab
# <file system>                 <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/sda1 during installation
UUID=89cb8d0c-200e-4022-8746-18603304d2c8 /      ext4    errors=remount-ro 0       1
# swap was on /dev/sda2 during installation
UUID=fd0b068b-ca58-4bef-8772-468f6c21c441 none   swap    sw              0       0

Setup netatalk

Apple afp network protocol. It needs two changes:

# file /etc/netatalk/AppleVolumes.default
# change "Home Directory" to "home_dir72", the blank char. gives a problem
# and for computer to computer copying it needs an address (72).

# file /etc/netatalk/afpd.conf
# http://ubuntuforums.org/showthread.php?t=1968048
# append last line
--tcp -noddp -uamlist uams_guest.so,uams_dhx2_passwd.so -nosavepassword -setuplog "default log_info /var/log/afpd.log" -mimicmodel RackMac

$ sudo /etc/init.d/netatalk restart
or
$ sudo service netatalk restart

But the practice shows, that Mac OS X will place a lot of invisible files (e.g. .DS_store) in all folders. So, it is better to use the smb protocol (smb://IP-address).

Installation Troubleshooting

Unfortunately after installation the boot ended in a message:

grub >

# tried to load the kernel
$ linux /vmlinuz
Error: Invalid magic number

So, I booted an Ubuntu 12.04 desktop live persistant version (USB-Stick) and tried to mount the RAID1 array.

# Install mdadm, RAID support
$ sudo apt-get install mdadm

# no email support (postfix)

# reboot

# assemble the software RAID1 array
$ sudo mdadm --assemble /dev/md0 /dev/md/0

# check for function - OK
$ cat /proc/mdstat
Personalities: [raid1]
md0 : active raid1 sdc1[0] sdd1[1]
      243158648 blocks super 1.2 [2/2] [UU]

# create a folder
$ cd /media
$ sudo mkdir raid

# mount RAID1 array
$ sudo mount -t ext4 /dev/md0 raid

Now install Boot-Repair and run, see Links. The program requests to deinstall mdadm because it interferes with DMraid.

Next step was to remove the internal 8 GB SSD, and try again Boot-Repair, started from an Ubuntu 12.04 desktop live USB-Stick.

This time Boot-Repair proposes for boot md0, but wants to install GRUB on dev/sda, which is the USB-Stick. There was no choice of selecting /dev/sdb or /dev/sdc of the software RAID1 array. The friendly developer Yann prepared a new version (2012-07-04) for that case, but unfortunately it did not work either.

Program gparted showed a partition /dev/md0p1 on /dev/md0.

The boot menu of Ubuntu 12.04 Alternate shows Boot from first harddisk, but when you select that from an Ubuntu on USB-Stick (flash memory), it will start itself, which is probably not what you want. So you could extend the menu.

After installing Ubuntu 12.04 Alternate on to a 8 GB SD-card, I tried to install GRUB in the RAID1 array with the program Boot-Repair (version 0.61-git 2012-07-04). It generates a very nice protocol in http://paste.ubuntu.com/1079212. Unfortunately it installs as the boot device the UUID of the SD-card. Even in the advanced mode, a target selection of md127 did not work.

Next step was to install GRUB2 (ver. 1.99) with chroot from the SD-card installation (sda1).

# RAID1 array /dev/md127

$ cd /media
# make folder for chroot
$ sudo mkdir raid1

# mount RAID1 array
$ sudo mount -t ext4 /dev/md127 raid1

# chroot to the RAID1 array
$ sudo mount --bind /dev raid1/dev
$ sudo mount --bind /proc raid1/proc
$ sudo mount --bind /pts raid1/pts
$ sudo mount --bind /sys raid1/sys
$ sudo chroot raid1

# no grub folder found
$ ls /boot

# install GRUB
$ apt-get install grub-pc
# select devices /dev/sdb, /dev/sdc (/dev/md127 did not work)
Ctrl+D (to exit out of chroot) 

# reboot
# It works :-)

Installation Boot Test

From the BIOS I can boot either hard disk /dev/sdb or /dev/sdc and RAID1 works. Device /dev/sda is an internal 8 GB SSD, which is not used. I did not boot with one hard disk only, because afterwards I have to wait 3 hours for resync.

# check status of RAID
$ cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sdc1[1] sdb1[0]
      243156856 blocks super 1.2 [2/2] [UU]

Setup of fail2ban

Fail2Ban is an intrusion prevention framework written in the Python programming language. It works by reading SSH, ProFTP, Apache logs etc.. and uses iptables profiles to block brute-force attempts.

For the moin wiki I have added a new filter apache-newaccount.conf in order to ban computers, which try to open an new account, which is anyway blocked, but cost resources.

For more detailed indormation, please see in the links.

To install fail2ban, type the following in the terminal:

$ sudo apt-get install fail2ban 

This will install:
python-central all 0.6.17ubuntu2 [41,4 kB]
fail2ban all 0.8.6-3 [84,1 kB]
libgamin0 i386 0.1.10-4ubuntu0.1 [18,6 kB]
gamin i386 0.1.10-4ubuntu0.1 [47,0 kB]
python-gamin i386 0.1.10-4ubuntu0.1 [8.140 B]

To configure fail2ban, make a 'local' copy the jail.conf file in /etc/fail2ban and edit

$ cd /etc/fail2ban
$ sudo cp jail.conf jail.local 

# Edit with mc:
# ignoreip: /24 meand mask 255.255.255.0 -> allow last segment 
ignoreip = 127.0.0.1/8 192.168.17.1/24
# bantime in seconds: 604800 -> 7 days = 168 h
bantime = 604800

# ACTIONS, global, especially for SSH, please see at the Links
banaction = iptables-allports
# if multiport is used, the attacker can try also on other ports

[ssh-ddos]
#enabled = false
enabled = true

[apache]
#enabled = false
enabled = true

[apache-multiport]
#enabled = false
enabled = true

[apache-noscript]
#enabled = false
enabled = true

[apache-overflow]
#enabled = false
enabled = true

[postfix]
#enabled = false
enabled = true

[sasl]
#enabled  = false
enabled = true

# In order to get the new parameters activated do:
$ sudo service fail2ban restart

Check for an installed iptables package:

$ which iptables
/sbin/iptables

or more verbose:

$ apt-cache policy iptables
iptables:
  Installiert: 1.4.12-1ubuntu4
  Kandidat:    1.4.12-1ubuntu4
  Versionstabelle:
 *** 1.4.12-1ubuntu4 0
        500 http://de.archive.ubuntu.com/ubuntu/ precise/main i386 Packages
        100 /var/lib/dpkg/status

To test fail2ban, look at iptable rules:

# use option -n, otherwise host name lookup takes a long time
$ sudo iptables -L -n

# check for the regexpr in the filter, example
$ fail2ban-regex /var/log/apache2/error.log /etc/fail2ban/filter.d/apache-auth.conf

Blocking IP-numbers which try to setup a new account in the moin wiki. I had about 5100 attacks within 6 days. It is not easy to figure out the failregex setup. You have to look up the already provided setup's and the log entry line you want to filter.

# Edit a new file in /etc/fail2ban/filter.d/

$ cat filter.d/apache-newaccount.conf
# Fail2Ban configuration file
#
# Author: RudolfReuter
#
# $Revision$
#

[Definition]

# Option:  failregex
# Notes.:  regex to match the password failure messages in the logfile. The
#          host must be matched by a group named "host". The tag "<HOST>" can
#          be used for standard IP/hostname matching and is only an alias for
#          (?:::f{4,6}:)?(?P<host>[\w\-.^_]+)
# Values:  TEXT
#
failregex = ^<HOST> .*action=newaccount

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#
ignoreregex =

# Add to the file "jail.local" in /etc/fail2ban
[apache-newaccount]

enabled = true
port     = http,https
filter   = apache-newaccount
logpath  = /var/log/apache*/access.log
maxretry = 2

# reload fail2ban config file
$ sudo /etc/init.d/fail2ban reload

# check for running
$ sudo /etc/init.d/fail2ban status
 * Status of authentication failure monitor   
 *  fail2ban is running

# check the number of attacks in the apache2 log
$ cat /var/log/apache2/access.log | grep newaccount | wc -l
5142

TYPO3 _cli_dispatcher cron logging

Since version 4.5.x of TYPO3 there is an extension Miscellaneous/Scheduler (ver.1.1.0). It needs an user _cli_scheduler with a dummy password. This allows to run some tasks in a regular interval, e.g. indexer.

While installing TYPO3, a cron job is setup for this scheduler, which runs every 5 minutes. This fills up the syslog.log file.

To keep it out of the syslog file there are two alternatives:

Please see below how to do so. My first try it do disable all cron logging.

# Edit file /etc/rsyslog.d/50-default.conf

# add "cron.none" to the following line
*.*;auth,authpriv.none,cron.none        -/var/log/syslog

# in case you want to have the cron log in an own file, uncomment the following line
#cron.*                         /var/log/cron.log

# activate configuration changes
$ sudo service rsyslog restart

List of pages in this category:

-- RudolfReuter 2012-07-03 03:47:47


Go back to CategoryServer or FrontPage ; KontaktEmail (ContactEmail)

ServerUbuntu1204 (last edited 2014-07-19 13:21:34 by RudolfReuter)