Linux & Sysadmin on PHP Boy Scout

Migrating away from Mediawiki and how to export its data

Wed, 19 Aug 2020 00:00:00 +0000

I like Mediawiki, it is a simple tool capable of doing a lot and can be very flexible and easy to customise. However its not always the right solution! I had a situation where we needed to migrate away from using it for a combination of security and usability reasons. So I thought it would be good to document it.

After reviewing a few things it was decided to move things over to the companies already existing O365 SharePoint as a new site. This sounded laborious as first, but actually turned out to be pretty straight forward.

We start with getting data out of Mediawiki, thankfully we only wanted the most recent revision and not the full history of a page. We use PostgreSQL as a back-end so it was reasonably straight forward to figure out how to extract the data in a sensible query.

SELECT 
 page_id as id, 
 page.page_title as title, 
 pagecontent.old_text as content, 
 page_touched as edited
FROM mediawiki.page
LEFT JOIN mediawiki.slots ON page.page_latest = slots.slot_revision_id
LEFT JOIN mediawiki.content ON content.content_id = slots.slot_content_id
LEFT JOIN mediawiki.pagecontent ON pagecontent.old_id = CAST(OVERLAY(content.content_address placing '' from 1 for 3) as integer)
ORDER BY page_touched DESC;

It tool a little sleuthing to realize that the slots table was the pivotal in extracting the latest page version. With the right join and a little mangling of the content_address field from the contents table to remove the “tt:” from the value and convert to an integer we now have a result set of all the page names and the latest revision of that page. I also added in the date the page was last updated to allow me to see when it was last edited as it was a live system migration and helped me to ensure content remained sync while both were still in play.

Once I had the query it was a simple job of writing an “Exporter” using Go Lang to extract the data and write it to files, I’ll chuck a snippet of code at the bottom of the post for you.

Mediawiki uses wikitext as a format so I needed to convert it to something more widely understood. Having used Pandoc in the past successfully I plumped for this as I knew it would handle a lot of options and was simple to use to convert to the markdown_mmd format

I Installed it via the ubuntu apt package available on my system and wired this in as a hacky exec command into my script… and voila! I had hardcopies of all the Mediawiki pages on my system in both wikitext and markdown_mmd format.

Why markdown_mmd I hear you ask… mainly because it gave me the cleanest conversion for use with the new markdown web page widget for Sharepoint’s modern interface.

Now we have the files we could do a little munging and parsing to convert URLs into the format needed for the new location in Sharepoint, easily done with a bit of regex pattern matching, which I wont go into as yours will be very different from mine… suffice to say looking for "wikilink" in my regex helped massively in finding all the occurrences I needed to update. I used sed but you could use whatever tool you like or add it into your version of the exporter

'SysAdmin/(.+) "wikilink"'

and with a little back referencing to substitute the values we need to keep and its all good.

Next came the import of the data into Sharepoint, but that is a post for another day.

package data

import (
	"bytes"
	"fmt"
	"github.com/jmoiron/sqlx"
	"github.com/rs/zerolog/log"
	"io/ioutil"
	"os"
	"os/exec"
	"path/filepath"
	"time"
	"wiki-export/src/util"
)

type Page struct {
	Id int
	Title string
	Content string
	Edited time.Time
}

type Exporter struct {
	Config util.ExporterConfig
	DB *sqlx.DB
}

func (l *Exporter) Export() {
	stmt := `
		SELECT page_id as id, page.page_title as title, pagecontent.old_text as content, page_touched as edited
		FROM mediawiki.page
		LEFT JOIN mediawiki.slots ON page.page_latest = slots.slot_revision_id
		LEFT JOIN mediawiki.content ON content.content_id = slots.slot_content_id
		LEFT JOIN mediawiki.pagecontent ON pagecontent.old_id = CAST(OVERLAY(content.content_address placing '' from 1 for 3) as integer)
		ORDER BY page_touched DESC
		;`

	page := Page{}
	rows, err := l.DB.Queryx(stmt)
	util.CheckErr(err)

	for rows.Next() {
		util.CheckErr(rows.StructScan(&page))
		wikiFilename := fmt.Sprintf("%s.mediawiki",filepath.Base(page.Title))
		mdFilename := fmt.Sprintf("%s.md",filepath.Base(page.Title))
		path := filepath.Dir(page.Title)
		wikiDir := fmt.Sprintf("%s/mediawiki",l.Config.TargetDir)
		mdDir := fmt.Sprintf("%s/%s",l.Config.TargetDir, l.Config.TargetFormat)

		if path != "." {
			wikiDir = fmt.Sprintf("%s/mediawiki/%s",l.Config.TargetDir , path)
			mdDir = fmt.Sprintf("%s/md/%s",l.Config.TargetDir , path)
		}
		util.CheckErr(os.MkdirAll(wikiDir, 0777))
		util.CheckErr(os.MkdirAll(mdDir, 0777))

		wikiTarget := fmt.Sprintf("%s/%s", wikiDir, wikiFilename)
		mdTarget := fmt.Sprintf("%s/%s", mdDir, mdFilename)

		log.Debug().Msgf("%s => %s -> %s", page.Title, wikiTarget, mdTarget)

		c := []byte(page.Content)
		util.CheckErr(ioutil.WriteFile(wikiTarget, c, 0777))

		cmd := exec.Command("pandoc", "-f","mediawiki", "-t", l.Config.TargetFormat, wikiTarget)
		
		var errorBuffer bytes.Buffer
		var outputBuffer bytes.Buffer
		cmd.Stdout = &outputBuffer
		cmd.Stderr = &errorBuffer
		err := cmd.Run()
		if err != nil {
			log.Err(err).Msgf("ERROR: %s", errorBuffer.String())
			util.CheckErr(err)
		}

		util.CheckErr(ioutil.WriteFile(mdTarget, outputBuffer.Bytes(), 0777))
	}
}

Adding Ambient Sounds to your Discord Server On LInux

Tue, 30 Jun 2020 00:00:00 +0000

I’m a Dungeon Master! I don’t mean that in the S&M sense! As in the game Dungeons & Dragons (https://dnd.wizards.com), where I run a weekly game as well as take part in a couple of campaigns as a player. It’s a lot of fun and something I would definitely recommend you have a go at if you are so inclined

There is a vast amount of tooling & tech out there that allows you to play remotely such as Virtual Table Tops, Character builders, online resources, etc. One such tool that gets used quite often is a chat service called Discord (https://discord.com) It’s really useful and allows you to easily be part of and manage communities of people…. Think IRC & Slack, but more up to date than IRC and less “workish” than Slack.

As part of my online games I like being able to have ambient music to match the surroundings the players are traveling through, as well as some active elements thrown in for good measure. This is possible in a few different ways using discord but the way I want to set it up can be somewhat frustrating to set up. Let me explain:

I have taken a shine to two tools in particular… Syrinscape (https://syrinscape.com) and Table Top Audio (https://tabletopaudio.com). The former being a windows app with an nice interactive mixing UI that allows you to combine and generate unique sounds, the latter being a lovely web service that has some fantastic loop-able ambient background tracks all 100% free.

I am wanting to be able to pipe the audio from these two services into my Discord server so that I can make use of the fantastic audio they offer. This is the journey of how I managed to get this working, partly as a reminder for me if I ever need to do this again and also to help others that may be looking to do the same.

My Setup

I’ve been a big fan of Ubuntu for a number of years, but since 20.04 I’ve found that the shine I’ve had for it has waned significantly. I wont go into the why and wherefore of it but I’m now running the excellent Pop_OS! from System76 (https://pop.system76.com) its an Ubuntu variant but with the bits I dislike removed. So assume that anything I’m doing is compatible with Ubuntu 20.04.

The Requirements

The ideal solution should see me being able to have a single instance of discord running that allows me to still use my mic to be able to talk, and to have my selected background playing with the ability to control the volumes of both the mic and the background independently.

Finding A Solution

A lot of googling led me to realise that there isn’t a perfect solution to fit my brief. The hardest part being not actually knowing what to google and a lot of the terminology being somewhat foreign to me as I’m not much of an audio engineer. However I finally stumbled upon a blog post by Emma Anderson dated June 2016 and thankfully it gives me a lot of the heavy lifting that I needed along with some explanation of what I’m trying to achieve, though I’m hopefully going to be more verbose here in what this all means and how it works.

PulseAudio

The first thing we need to do is make sure the packages for pulseaudio and pavucontrol are installed. These will allow us to manipulate the way we capture sound and redirect it to the appropriate inputs and outputs.

apt install pulseaudio pavucontrol

Virtual Input & Virtual Mic

What we are going to try to achieve, is to create two new elements inside of Pulseaudio;

a Virtual input that we can channel the applications creating our background sounds which will allow us to control the volume independently.
a Virtual Microphone that we can channel our both our normal microphone and the new Virtual input into.

By creating these elements we can then use the pavucontrol tool to select what needs to be redirected where. so lets get started.

pactl load-module module-null-sink sink_name=VirtualInput
pacmd update-sink-proplist VirtualInput device.description=VirtualInput
pacmd update-source-proplist VirtualInput.monitor device.description=VirtualInput.monitor

Here we have two commands, the first will create our new Virtual Input as what is referred to as a “null sink”. This on its own is not really very useful for us as we also need what is referred to as a “source”, thankfully when we run this command it also created a new “source” for us.

On it’s own that should be more than enough, but running the 2nd & 3rd command makes our live a lot easier because it will apply some very useful labels to both of the newly created sink and source. In this case VirtualInput for the sink and VirtualInput.monitor for the source. Having these in place makes it a lot simpler to configure things with pavucontrol.

Next we need to create our Virtual Mic using some very familiar looking commands.

VM=$(pactl load-module module-null-sink sink_name=VirtualMic)
pacmd update-sink-proplist VirtualMic device.description=VirtualMic
pacmd update-source-proplist VirtualMic.monitor device.description=VirtualMic.monitor

again we have now created a new new pair of sink and source with some nice easy to recognise labels that we will use when we start working with pavucontrol.

The next piece to our puzzle is creating the components that will let us define a connection from the VirtualInput and our physical microphone to the newly created VirtualMic. We do this with two identical commands;

pactl load-module module-loopback sink=VirtualMic
pactl load-module module-loopback sink=VirtualMic

we now have most of the elements that we need to configure everything to work.

Listening to my own Ambience

Before we can start wiring it all together we need to ensure we can also listen back to our own ambience. This involves us creating one more “loopback” module that points to the speakers we are wanting to listen to. Lets find out what our options are by running;

pacmd list-sinks | awk '/index:/ {print $0}; /name:/ {print $0}; /device.description/ {print $0}

This lists all of the available “sinks” that we can use. on my daily driver laptop I get;

 * index: 1
	name: <alsa_output.pci-0000_00_1f.3.analog-stereo>
		device.description = "Built-in Audio Analogue Stereo"

This tells us the “index” for the device, its name and also some kind of description. The important bit for us here is the name as we will need that to create our new “loopback” with the command;

pactl load-module module-loopback sink=alsa_output.pci-0000_00_1f.3.analog-stereo

This creates the last piece for our puzzle!

Connecting it all together

I’m now going to assume you have logged yourself into the Discord client and fired up your copy of Syrinscape… but you should just as easily swap out these for something else of your choice.

Now we can start pavucontrol either from the command line or you can look for it in your applications menu. Once it loads you will hopefully be presented with something that looks like;

For this next step I am specifically starting on the “Recording” tab of pavucontrol this is to allow us to set up what is going to be captured. I have updated the drop-down at the bottom left to show “All Streams” as this will make it quicker to configure… Starting at the top we have two entries for;

Loopback to VirtualMic from: These are the result of the first two “loopback” modules we created with the pactl command we ran previously. They are going to allow us to capture the audio streams from our physical microphone, mine here is the TONOR TC-777 and our newly created VirtualMic.

followed by a single entry for;

Loopback to Built-in-Audio Analougue Stereo from: which is the last “loopback” module that we create to let us hear our own Ambience, Having this set to our VirtualInput means that anything that pipe into our VirtualInput will also come out of our speakers.

and finally;

WEBRTC VoiceEngine: Once you connect to a voice channel in discord this will appear and it allows us to specify which of our devices it should be reading the audio feed from. For our purposes we have this set to our VirtualMic so that we can have our mixed audio feeds

Now that recording is configured we can sort out our playback.

Here we can see the “Playback” tab of pavucontrol, again set to show “All Streams”. This time I’m going to run through the elements here starting from the bottom of the list and working my way up…

WEBRTC VoiceEngine: This again is our connection to a Discord voice channel, as you can see I have this set to play back all of its output via Built-in Audio Analogue Stereo which is how my Operating system has labelled my physical speakers.

Syrinscape.exe: This is the Syrinscape application, that I run through PlayOnLinux (https://www.playonlinux.com), and I will use to generate all of my lovely ambient sounds. This is set to play all of its “audio stream” on our VirtualInput.

The next two items in our list, providing you have configured the Recording tab first should look as in the image. Changing the “Loopback to VirtualMic” entries on the Recording tab will change the labels of these two entries.

Loopback of VirtualInput.monitor on: it seems we have two of these entries and where we can tell the to pip all of the audio we are now capturing on our VirtualInput In this case we want it to go to two places, our VirtualMic so that it can be sent to both our Discord audio channel and also to our Built-in Audio Analogue Stereo speakers.

Loopback of Built-in Audio Analogue Stereo on: is where we now direct the input from our physical microphone and feed that straight into our VirtualMic.

The other entries in the list here are for firefox and the system itself and are not relevant to what we are trying to achive.

Winner Winner Chicken Dinner

That’s is effectively all we need to do… From here on in anything you play via the Syrinscape app will be merged with your microphone input and passed to Discord. You can then use the volume sliders in pavucontrol to adjust the levels of all the inputs to suit your own personal preference.

Though I will make a few small suggestions about how to configure your discord settings. You shouldn’t need to make any adjustments to the input and output devices which should now be set to Default as your “Input Device” if you change this it will override the changes we have made and you will need to go back to the Recording tab of pavucontrol and switch WEBRTC VoiceEngine back to InputMic , but…

I would recommend disabling automatic input sensitivity and lowering the sensitivity slider all the way down to -100dB… this is to allow for the potential low and subtle tones and ambient elements you may want to play… be warned though it makes it very very easy for an low quality microphone (such as the Built-in Audio Analogue Stereo microphone found on my laptop) to pick up other noises such as your systems fans, mouse clicks and typing. A simple way to combat this is to get a reasonable quality external cardioid condenser microphone which eliminates a lot of this unwanted background.

One last thing

That should be it for now… I’ll leave you with one final thing. This is a simple little bash script I threw together that I run in can a terminal to create all the components and if I want will then clean them all up and remove them. If you really want you could set it up as a permanent implementation, but I’ll let you google for that solution!

#!/bin/bash

LB1=""

listenback() {
 echo ""
 echo "Listing all possible output devices"
 pacmd list-sinks | awk '/index:/ {print $0}; /name:/ {print $0}; /device\.description/ {print $0}'
 echo ""
 echo "Please enter the name of the output device to create a loopback for (leave blank to skip): "
 read S1
 if [ "$S1" != "" ]; then
 	echo " * Creating Loopback for '$S1'"
 	LB1=$(pactl load-module module-loopback sink="$S1")
 fi
}

cleanup() {
 while true; do
 read -p "Finished? do you want to clean up and remove modules [Yn]: " yn
 case $yn in
 [Yy]* ) return 0;;
 [Nn]* ) return 1;;
 * ) echo "Please answer yes or no.";;
 esac
 done
}

listenback

echo " * Creating VirtualInput"
VI=$(pactl load-module module-null-sink sink_name=VirtualInput)
pacmd update-sink-proplist VirtualInput device.description=VirtualInput
pacmd update-source-proplist VirtualInput.monitor device.description=VirtualInput.monitor

echo " * Creating VirtualMic"
VM=$(pactl load-module module-null-sink sink_name=VirtualMic)
pacmd update-sink-proplist VirtualMic device.description=VirtualMic
pacmd update-source-proplist VirtualMic.monitor device.description=VirtualMic.monitor

echo " * Creating loopbacks for VirtualMic"
VML1=$(pactl load-module module-loopback sink=VirtualMic)
VML2=$(pactl load-module module-loopback sink=VirtualMic)


echo "All modules have been loaded have been configured! Run pavucontrol to configure your devices."

if cleanup; then
 pactl unload-module "$VML2"
 pactl unload-module "$VML1"
 pactl unload-module "$VM"
 pactl unload-module "$VI"
 if [ $LB1 != "" ]; then
 pactl unload-module "$LB1"
 fi
 echo "All modules have been unloaded"
else 
 if [ $LB1 != "" ]
 then
 echo "Modules $LB1, $VI, $VM, $VML1 & $VML2 remain loaded"
 else
 	echo "Modules $VI, $VM, $VML1, & $VML2 remain loaded"
 fi
fi

Encrypting additional drives with LUKS on Linux

Mon, 29 Jun 2020 00:00:00 +0000

Encryption is king nowadays with everyone having mobile devices. We have a significant number of people on laptops that travel around and also workstations that live in open plan offices. This means we encrypt all of our disks… just in case. 99% of the time is super simple to do as most OS installers give you the option to do it, some now ven enforce it as a default option. This post however is about adding an additional disk to the system and making it automatically mount on system startup.

So let me set the scene, we have a data-scientist that’s running out of disk space for a task they are running on their Ubuntu 18.04 Workstation. At some point the workstation had an upgrade to the HDD in the past to a shiny new SSD, and the old 4Tb spinning disk was left in the chassis that they want to use for this very specific task.

Now this workstation has been through a couple of data-scientists over the last 12 months and unfortunately the LUKS password that had been set up for the old spinning disk has gone walkabouts… so the plan is as follows

flatten the old disk and set up a new partition using the whole disk
generate a new secure encryption key
set up LUKS encryption on the new partition
use Ext4 as a filesystem
enable auto decryption of the disk
add the new partition to the fstab to mount on system startup

N.B Assume that we are running everything as the root user

Flatten the disk

As we cant recover anything we are going to flatten the disk using parted (apt install parted to install) to allow is to create a partition greater than 2Tb, but first we are going to identify the disk we are working with… I tend to favour using either fdisk -l or as a more concise option lsblk -p which gives us a an easy to interpret overview something like:

/dev/sda 8:0 0 1.8T 0 disk 
├─/dev/sda1 8:1 0 512M 0 part /boot/efi
├─/dev/sda2 8:2 0 732M 0 part /boot
└─/dev/sda3 8:3 0 1.8T 0 part 
 └─/dev/mapper/sda3_crypt 253:0 0 1.8T 0 crypt 
 ├─/dev/mapper/ubuntu--vg-root 253:1 0 1.8T 0 lvm /
 └─/dev/mapper/ubuntu--vg-swap_1 253:2 0 976M 0 lvm [SWAP]
/dev/sdb 8:16 0 3.7T 0 disk

I can tell from this that we are looking at using the disk that is currently at /dev/sdb and its showing as being 3.7Tb in size.

Great… now to set up our new partition using the command parted /dev/sdb which gives us an interactive shell to work with (you can see the prompts in the output below are prefixed with (parted)

GNU Parted 3.2 
Using /dev/sdb 
Welcome to GNU Parted! Type 'help' to view a list of commands. 
(parted) mklabel gpt

The command mklabel gpt will wipe the partition table for /dev/sdb and give us a clean slate to work from

(parted) unit TB

We now set parted to think in Terabytes as the default reference size using the command above.

(parted) mkpart primary 0.00TB 3.70TB

Now we get to create the actual partition. You can see from the command above that we are using the command mkpart and telling it to create a primary partition type.

(parted) print 
Model: ATA WDC WD4005FZBX-0 (scsi) 
Disk /dev/sdb: 4.00TB 
Sector size (logical/physical): 512B/4096B 
Partition Table: gpt 
Disk Flags: 
 
Number Start End Size File system Name Flags 
 1 0.00TB 4.00TB 4.00TB primary

We can chek everything went smoothly using the print command which gives us confirmation that a new primary partition is present. We can now leave parted with a simple.

(parted) quit

And we can now use fdisk -l or lsblk -p to see that we now have a partition waiting for us at /dev/sdb1.

/dev/sda 8:0 0 1.8T 0 disk 
├─/dev/sda1 8:1 0 512M 0 part /boot/efi
├─/dev/sda2 8:2 0 732M 0 part /boot
└─/dev/sda3 8:3 0 1.8T 0 part 
 └─/dev/mapper/sda3_crypt 253:0 0 1.8T 0 crypt 
 ├─/dev/mapper/ubuntu--vg-root 253:1 0 1.8T 0 lvm /
 └─/dev/mapper/ubuntu--vg-swap_1 253:2 0 976M 0 lvm [SWAP]
/dev/sdb 8:16 0 3.7T 0 disk 
└─/dev/sdb1 8:17 0 3.7T 0 part

Generating an encryption key

Our disk is now ready for use, but not yet encrypted, so our next step is to create a key that can be used when we encrypt the disk. As we are going to be mounting it automatically we want to use a keyfile to store the key. You can of course create a key by mashing the keys on the keyboard, but I tend to prefer letting something else do the hard part for me.

First we create somewhere to store the key… I opted for,

mkdir -p /etc/crypt/keys

But feel free to put it wherever you want just as long as its only accessible by the root user. Next we generate the keyfile using the command:

dd bs=512 count=4 if=/dev/urandom of=/etc/crypt/keys/sdb1 iflag=fullblock

Here I am using /dev/urandom as my randomness generator, but you could use any valid generator of your choice. With this set of parameted dd will read the stream of “randomeness” and write 2048 bytes to our keyfile at /etc/crypt/keys/sdb1. If you want to be a little more complex about teh size and shape of your key then have a look at https://man7.org/linux/man-pages/man1/dd.1.html

Encrypting the Disk

Hopefully it will already be installed because you encrypted your root disk at installation, but if not you can run apt install cryptsetup to get going.

The command to do the encryption is actually very simple.

cryptsetup luksFormat /dev/sdb1 /etc/crypt/keys/sdb1

You can see that using the cryptsetup tool we are asking it to execute teh command luksFormat but while it says format in the command this is a little misleading as it doesn’t actually format the disk but just rewrites a portion of bytes at the beginning of the partition to enable encryption. we then tell it the partition we want encrypting, here its /dev/sdb1 and finally we pass in the keyfile we just generated and saved at /etc/crypt/keys/sdb1. If you omit the keyfile it will still encrypt teh disk but will prompt you to enter the key manually.

As soon as you press enter you will be warned of teh danager of what you are doing… so double check you are encrypting the right partition and follow the instructions that should look something like :

WARNING! 
======== 
This will overwrite data on /dev/sdb1 irrevocably. 
 
Are you sure? (Type uppercase yes): YES 
Command successful.

And that’s it… the disk is encrypted and ready to use. There are a few ways you can now work with the disk. the quickest and easiest is to just decrypt the disk manually using cryptsetup to open the disk.

cryptsetup open /dev/sdb1 sdb1_crypt -d /etc/crypt/keys/sdb1

Here we open /dev/sdb1 and give is a new name of sdb1_crypt and we unlock it using the -d argument to tell it the keyfile we generated before.

That is the dis decrypted and ready to roll… you can now use fdisk -l or lsblk -p to confirm that it is now available at /dev/mapper/sdb1_crypt.

/dev/sda 8:0 0 1.8T 0 disk 
├─/dev/sda1 8:1 0 512M 0 part /boot/efi
├─/dev/sda2 8:2 0 732M 0 part /boot
└─/dev/sda3 8:3 0 1.8T 0 part 
 └─/dev/mapper/sda3_crypt 253:0 0 1.8T 0 crypt 
 ├─/dev/mapper/ubuntu--vg-root 253:1 0 1.8T 0 lvm /
 └─/dev/mapper/ubuntu--vg-swap_1 253:2 0 976M 0 lvm [SWAP]
/dev/sdb 8:16 0 3.7T 0 disk 
└─/dev/sdb1 8:17 0 3.7T 0 part 
 └─/dev/mapper/sdb1_crypt 253:3 0 3.7T 0 crypt /mnt/4tb-1

This tells us that the newly decrypted disk is now available at /dev/mapper/sdb1_crypt and is a volume of 3.7Tb… Exactly what we were hoping for!

All finished with your encrypted disk… you can just as easily close it again using:

cryptsetup close sdb1_crypt

Setting up the Filesystem

Ok, we have an encrypted partition, we can decrypt it but we cant mount it yet as we don’t have a file system to work with. Let’s take care of that real quick by opening up the partition again.

cryptsetup open /dev/sdb1 sdb1_crypt -d /etc/crypt/keys/sdb1

And now that its available we are going to set up an ext4 filesystem using the command mkfs.ext4 /dev/mapper/sdb1_crypt which, all going according to plan, should look something like:

mke2fs 1.44.1 (24-Mar-2018) 
Creating filesystem with 976753664 4k blocks and 244195328 inodes 
Filesystem UUID: d797be67-c53e-49d3-897e-c624b21a22d3 
Superblock backups stored on blocks: 
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
 102400000, 214990848, 512000000, 550731776, 644972544 
 
Allocating group tables: done 
Writing inode tables: done 
Creating journal (262144 blocks): done 
Writing superblocks and filesystem accounting information: done

we are now good to go… lets try mounting the filesystem with

mount -t ext4 /dev/mapper/sdb1_crypt /mnt

I’m just mounting straight to /mnt but obviously this can be any folder you want. If the command worked we can easily confirm it with a quick df -h:

Filesystem 1K-blocks Used Available Use% Mounted on
udev 32846708 0 32846708 0% /dev
tmpfs 6578140 2308 6575832 1% /run
/dev/mapper/ubuntu--vg-root 1919562064 993193376 828790500 55% /
tmpfs 32890688 200 32890488 1% /dev/shm
tmpfs 5120 4 5116 1% /run/lock
tmpfs 32890688 0 32890688 0% /sys/fs/cgroup
/dev/sda2 721392 276068 392860 42% /boot
/dev/sda1 523248 6232 517016 2% /boot/efi
/dev/mapper/sdb1_crypt 3844637680 0 3844637680 1% /mnt

Excellent… you can now start working with your new partition… however lets un-mount and close the drive quickly with a

umount /mnt && cryptsetup close sdb1_crypt

And then we can move onto…

Automatic Decryption

This is a lot simpler that you may realise… all we need to do is add a new line to the file /etc/crypttab! But first we need one last piece of information we don’t yet have, but we can easily get with the command

sudo cryptsetup luksDump /dev/sdb1 | grep "UUID"

This will use luksDump to get information about the encrypted partition and then uses grep to specifically target the property UUID which we will need to identify the partition in the next step.

Now in your favourite editor of choice add the following line, replacing the spoof UUID here with the one we just found.

sdb1_crypt UUID=1111111111-2222-3333-4444-555555555555 /etc/crypt/keys/sdb1 luks

Here we are giving the decrypted volume a unique label for teh decrypted label to be made available at the appropriate /dev/mapper/* location. We also specify the UUID to identify the partition to decrypt… we could use the path /dev/sdb1 but using the UUID is more explicit and prevents any confusion if another partition happens to present itself as /dev/sdb1 at some point in the future. Third we have the path to our newly generated keyfile and finally we have the encryption mode that we are using for encryption which here is luks. For more info on crypttab have a look at https://www.freedesktop.org/software/systemd/man/crypttab.html

We can now test that auto decryption is working using:

cryptdisks_start sdb1_crypt

which if successful should have an output like:

 * Starting crypto disk... 
 * sdb1_crypt: INSECURE MODE FOR /etc/crypt/keys/sdb1, see /usr/share/doc/cryptsetup/README.Debian. 
 * sdb1_crypt (starting).. 
 * sdb1_crypt (started)...

All that’s left to do now is set up

Auto-mount the filesystem

Hopefully we now are on really familiar ground… we can now treat /dev/mapper/sdb1_crypt as a bog standard ext4 partition that can be mounted via the /etc/fstab by adding the line:

/dev/mapper/sdb1_crypt /mnt ext4 defaults 0 2

As you can see its pretty ordinary, exactly as you would expect, obviously swapping out /mnt with the location of your choice to mount the filesystem. If you are not wholly familiar with fstab then its definitely worth having a look at https://help.ubuntu.com/community/Fstab as it gives a good overview for those who are new to it…

Finally we can check that it all works with:

mount -a

And at this point I am pretty sure I can hear the fat lady singing…

Connecting to Ubuntu 18.04+ using RDP

Mon, 20 May 2019 00:00:00 +0000

We have a mix of different setups that the Software Engineer and Data Scientists use to get their work done. There are some using just Linux on laptops, Some on MacBooks and some on the various versions of Windows.

For those not using Linux as their primary OS we have a bunch of Desktops that run Ubuntu 18.04+ for them to connect to. SSH can do quite a lot but a few of the team work remotely and in house we prefer RDP for that kind of thing rather than VNC.

We have had some issues with connections in the past so this post exists to remind me how next time I need to set it up. First we need to install the xRDP server package.

sudo apt install xrdp

Next we need to ensure that we have the right ports open on the workstation. If like me you also use UFW to manage your firewall rules then open port 3389 using…

sudo ufw allow 3389

The issue left, is that you will get an annoying pop up when you log in about a colour management profile needing to be set up and asking you to provide your password. Even then you may still get some annoying crash pop-ups.

I found a really good solution to this at http://c-nergy.be/blog/?p=12043 which I’ve cribbed and paraphrased below

Create the file /etc/polkit-1/localauthority/50-local.d/45-allow-colord.pkla (using your editor of choice and sudo)and add the following contents

[Allow Colord all Users]
Identity=unix-user:*
Action=org.freedesktop.color-manager.create-device;org.freedesktop.color-manager.create-profile;org.freedesktop.color-manager.delete-device;org.freedesktop.color-manager.delete-profile;org.freedesktop.color-manager.modify-device;org.freedesktop.color-manager.modify-profile
ResultAny=no
ResultInactive=no
ResultActive=yes

We now need to clear any crash dumps from the workstation

sudo rm /var/crash/*

You should then be good to connect to using whatever RDP client you prefer… I like Remmina myself but each to their own.

Dell DisplayLink D6000 & Ubuntu 18.04+ Issues

Tue, 14 May 2019 00:00:00 +0000

I love Ubuntu… I’m pretty fond of dell kit too!

So I was rather chuffed when I started working at Medicines Discovery Catapult because they let me have both. When you look at my desk it looks like it could be an advert for Dell. Laptop, monitors, dock, keyboard and mouse…. its great when you have a corporate account with a Dell reseller

However while I’ve had a lot of success with the D3000 DisplayLink dock on Ubuntu I found that I’m now having to deal with the upgraded D6000… which doesn’t play very nicely with the more recent versions of Ubuntu (we are talking 18.04 and later)

I kept finding that after a random amount of time the D6000 would randomly seem to power down… I would lose the screens, audio, networking and USB. and the only way I could fix it is to unplug it from teh laptop and plug it back in. Not ideal, especially if I’m in the middle of a video call or debugging something on the net

Being the kind of techie I am my first port of call checking my logs… but I couldn’t see anything that would cause this random disconnect. So off to google I went… eventually I found a lot of information telling me it was part of power management causing things to start powering down… In this case it implied that it was something trying to suspend USB… which sounded really plausible!

So a little more research suggested that I should be using laptop mode tools to disable the ability for USB to be suspended. I gave it a go, though I was dubious as in my mind I shouldn’t have needed to install an additional package (albeit a great one for tweaking your power management on a laptop running Linux)

Alas no joy! And I had too much to do to start debugging in depth and ripping apart other peoples code to figure it out.

What did I do? you ask. Well, I just put up with it for a few weeks, but gradually it began to grate on my nerves. However there was that one day where it didn’t turn off… and that left me perplexed… I checked if any updates had been applied in my last apt update && apt upgrade … nothing…. it then dawned on me that I had plugged in the headset I used for conference calling into the audio in/out on the dock instead of directly into the laptop.

Now I had a little more information I was able to deduce (with googles help) that the laptop was actually suspending USB, but that the trigger was actually pulseaudio. At this point it becomes really easy to solve the problem.

Solution

Edit /etc/pulse/default.pa using your preferred editor (and sudo)

Find the line

### Automatically suspend sinks/sources that become idle for too long
load-module module-suspend-on-idle

And comment it out and save!

Lastly, because its run as a user service you need to restart the Pulse Audio daemon using the command

systemctl --user restart pulseaudio.service

or you could just logout and back in again

Using Gmail aliases with Evolution

Wed, 06 Jan 2016 00:00:00 +0000

If your anything like me you have a large number of email aliases that you use with Gmail which is great. However I use Evolution as a mail client more often than not when using Gnome3 as a desktop.

It’s very easy to set up Evolution to create separate outbound email accounts that you can use for handling all of your aliases. It doesn’t yet support OAuth2 as an authentication mechanism for any account that is not set up using the built-in Gnome Online Accounts integration.

This is a real pain as Google have disabled the more common ‘plain’ and ’login’ authentication mechanisms for use with an SMTP only account. Meaning that any time that you try to connect to smtp.gmail.com:587 with STARTTLS you will get some form of error message to the effect of “Bad Authentication”.

Hopefully I’ll find a workaround at some point in the near future or Evolution will add the facility to enable OAuth2 as an available authentication mechanism.

In the mean time there is a workaround if you visit https://www.google.com/settings/security/lesssecureapps you can enable these less secure authentication mechanisms allowing you to once again connect and send email via email addresses using SMTP

Better Output for MySQL Select Command Using \G

Wed, 24 Apr 2013 00:00:00 +0000

If you ever find yourself using MySQL via command line and end up with something like this:

And thought there must be another way, well here it is: Use \G instead of ; at the end of your select command.

For example:

select * from CHARACTER_SETS\G

Below is an image of the output from this select:

Happy Querying!

Glorious Gluster - How to setup GlusterFS on Rackspace Cloud and Ubuntu 12.10

Fri, 15 Mar 2013 00:00:00 +0000

A few of our projects recently called for a distributed file-system that provided high availability and redundancy. After a tip off from a fellow techie and a quick browse around the net it appeared that a solution called GlusterFS appeared to tick all the boxes for what we were wanting.

However setting it up turned out not to be as trivial as I had originally anticipated. I’m going to try and put down the process we have evolved for setting it up on Ubuntu in the cloud

A couple of things to clear up first.

We are using Rackspace for our cloud but beyond the setup of the servers it should still be relevant
There are a number of ways to interact with Rackspaces set up but for this we are going to use the cloud control panel
We use Ubuntu as our preferred server which means that our config tends to be all over the place compared to other guides
You will need to set up a minimum of 2 servers and a separate block storage device for each.
We have set up and broken a few different variations of gluster setup so far and make no guarantees that the setup in this blog is infallable but its the best wehave so far.

Setting up the hardware

First things first. We are going to need to set up are some servers.

Feel free to create any size server you want. Just make sure to select Ubuntu 12.10 (or whatever version you may have that is newer).

You will also need to define a new network to work with. We use this to isolate the traffic between the nodes of our new gluster.

You can create a new network when creating the first of your servers. On the creation page under the networks heading you can find a “Create Network” button.

Hopefully this should be quite self explanatory. Now when you create subsequent servers you will then have the option to attach your new network (“GlusterNet” in my example).

Once the two starting nodes have been created then you need to add some additional block storage to store your data on. Make sure that you create blocks that have sufficient capacity for your needs. Something else to consider is using High Performance SSD storage. Its a little on the pricy side but well worth the expense if you are trying to eak out every ounce of performance from the implementation.

You will then need to attach one to each of your servers.

Once attached you will be able to see the details of the block mount point from the block storage details page.

Make a note of the mount point (in this case “/dev/xvdb”) as we will need that in a minute.

Prepare the Server

Now that we have a the hardware ready we can shell into a server to set it up.

First you need to shell into your server and update its OS as the images provided by most cloud supplier tends not to have the latest patches and updates. In our case it’s as simple as:

apt-get update 
apt-get upgrade

Once that’s done we then need to prepare the Block Storage device ( henceforth refered to as a “brick”)

if you run fdisk -l you should see that an entry that looks something like

Disk /dev/xvdb: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders, total 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/xvdb doesn't contain a valid partition table

This indicates that our brick needs a partition table and formatting. We can achieve this be doing the following

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xe7da4288.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Partition type:
 p primary (0 primary, 0 extended, 4 free)
 e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-209715199, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-209715199, default 209715199): 
Using default value 209715199

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

I’ve highlighted the prompts and my responses. All we are doing here is creating a default partition table that has a single partition which uses up the whole disk.

now running fdisk -l should give us something that looks like

Disk /dev/xvdb: 107.4 GB, 107374182400 bytes
43 heads, 44 sectors/track, 110843 cylinders, total 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xe7da4288

 Device Boot Start End Blocks Id System
/dev/xvdb1 2048 209715199 104856576 83 Linu

As you can now see we have a valid device of /dev/xvdb1 that we can mount_._ However we need to create a valid filesystem on the new brick before we can mount it. I have been doing this with Ext4 rather than XFS (which is the recommened filesystem from gluster), this is mainly down to the fact that when i tried using XFS I kept getting some issues with performance and access. I’m sure that with further investigation I could resolve this but as of yet haven’t had chance to. So far though I have had zero issues using Ext4. To create the filesystem we run:

mkfs.ext4 -j /dev/xvdb1

Next, create a folder to mount to, easily done by executing:

mkdir -p /glusterfs/brick

Finally, the simplest way to mount the device is via your /etc/fstab by adding the line

/dev/xvdb1 /glusterfs/brick ext4 defaults 1 2

and running mount -a as root (this will also mean that it mounts on boot for you automatically as well.)

Next we need to install the latest gluster version. At the time of writing this was v3.3.1. You can find a version to suit your OS at http://www.gluster.org/download. If you are using Ubuntu you can do the following

apt-get install software-properties-common
add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.3
apt-get update
apt-get install glusterfs-server glusterfs-client

By this point you will now have a single working server to continue on your going to need to set up your second server ready to create your new volume.

Once you have your second (or third, fourth, etc) setup its a good idea to add a reference to each one of them to your /etc/hosts file. This is not really necessary and you can just use the IP addresses of each server but it saves you having to remember each IP and makes it easier to identify.

Remember that we are going to be working with the new network interface you created earlier (i.e “GlusterNet”). to get the IP of your GlusterNet interface a quick ifconfig will show you an interface with an IP that matched the CIDR from earlier. In my case I now have 2 IPs of 192.168.3.1 & 192.168.3.2.

So now I add the following lines to my /etc/hosts file

192.168.3.1 gluster1
192.168.3.2 gluster2

Creating our volume

Now that the servers are prepared we can now play with the the tool gluster.This tool is a life saver in getting everything configured quickly and you can easily get a list of what its capable of by running gluster help. Now Im not going to take you through every command and option and would recomend reading the gluster manual to learn more.

What this tool actually does is help generate and manipulate all the required config that is then stored at /var/lib/glusterd/.

Firstly we need to tell gluster is that we have a pool of servers that will communicate with each other. Gluster refers to these as peers. To do this you need to run gluster peer probe gluster2 on each server for each server that will be used, replacing “gluster2” with the name names you defined in your /etc/hosts file. This will then create the appropriate files at /var/lib/glusterd/peers/

Now that all our peers have been defined we can get to actually creating the new distributed volume. This however requires a little consideration as there are some decisions you need to make.

If we take a look at the help for creating a new volume we can see that we need to decide on what options to use

volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT>] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>

- what are we going to name our volume
[stripe ] [replica ] - are we going to crate a striped or replicated volume and how many “bricks” are we going to create this volume with
[transport {tcp|rdma|tcp,rdma>] - What transport protocol do you want the peers to communicate with
- which servers/bricks do you want to use.

for more information on how to create you volume and what all the options mean have a look at these links

http://gluster.org/community/documentation/index.php/Gluster_3.2:_Configuring_Distributed_Replicated_Volumes

http://gluster.org/community/documentation/index.php/Gluster_3.2:_Configuring_Distributed_Striped_Volumes

for our purposes we are going to run

gluster volume create myvolume replica 2 transport tcp gluster1:/glusterfs/brick gluster2:/glusterfs/brick

This now creates a new volume that spans both of our servers. you can confirm that this is the case by running gluster volume info and you should get something that looks like

Volume Name: myvolume
Type: Replicate
Volume ID: d3dd24fd-9482-44c3-9503-24291fad8193
Status: Created
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gluster1:/glusterfs/brick
Brick2: gluster2:/glusterfs/brick

running this on both servers should give you the same results.

What you will now find is that the gluster command has created a plethora of files at /var/lib/glusterd/vols/myvolume/. As you work with gluster more and more you will find yourself drawn to these files as they control all the different aspects of how the volume works and performs. Most importantly we will need some information from these files when we come to configure a client to mount the volume.

All that is left to do now is start the volume which can be easily done with a quick gluster volume start myvolume

At this point we have now completed setting up our volume but we need to add some security. I would strongly recommend setting up a firewall using ufw to control access to the server. The easiest way to do this is to allow all traffic on your “GlusterNet” network interface as only the servers you attach to that network will have access. you can find a guide to using ufw at https://help.ubuntu.com/12.10/serverguide/firewall.html.

Mounting a Client

Now that we have a working volume we need to add some clients. To do this you will need to create a new server as above that is attached to the “GlusterNet” network but without the block storage (unless you really want it that is).

Make sure to add your gluster dfinitions to your /etc/hosts file

Once you have your new client server ready we can install the gluster client

apt-get install software-properties-common
add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.3
apt-get update
apt-get install glusterfs-client

I’ve seen a number of different guides that tell you to install glusterfs-server as well but I have as yet had no need to as it all works without it.

Now there are a lot of way that you can mount your new Gluster volume. I have tried a few and have had varying results. What I have found is that the best way is to create a volume file. To do this we create a new file at /etc/glusterfs.vol.

volume gluster1
 type protocol/client
 option transport-type tcp
 option remote-host gluster1
 option remote-subvolume /glusterfs/brick
 option username <username>
 option password <password>
end-volume

volume gluster2
 type protocol/client
 option transport-type tcp
 option remote-host gluster2
 option remote-subvolume /glusterfs/brick
 option username <username>
 option password <password>
end-volume

volume replicate
 type cluster/replicate
 subvolumes gluster1 gluster2
end-volume

volume writebehind
 type performance/write-behind
 option cache-size 1MB
 subvolumes replicate
end-volume

volume cache
 type performance/io-cache
 option cache-size 400MB
 subvolumes writebehind
end-volume

What you will notice is that there is a and required for this to work. You can find these details on one of your peer servers in the file /var/lib/glusterd/vols/myvolume/trusted-myvolume-fuse.vol.

This /etc/gluster.vol file is basically going to inform the gluster-client software about how to connect to the gluster volume and all the available nodes to connect to. This provides us with some level of fail-over so should one node become unavailable the gluster client will seamlessly switch to a different one. It also allows us to define additional “translators” such as the performance-io one that you can see here. I would strongly recommend reading through the available translators to see which may be useful to you.

Now one of the main issues you will find with Ubuntu is that it will fail on boot if you try to add this mount to your fstab. To get around this you can use Upstart. if you create the following file at /etc/init/glusterfs-mount.conf making sure to change to the interface for your GlusterNet network (i.e. eth0 or eth1 or eth2, you get the idea)

author "Matt Cockayne"
description "Mount GlusterFS after networking available"

start on net-device-up IFACE=<interface>
stop on stopping network
stop on starting shutdown

script
    mount -t glusterfs /etc/glusterfs.vol /glusterfs
end script

As you can see we are using a straight mount command. The magic is that this will not be executed until the start clause validates which in this case is not until the network interface for “GlusterNet” is up and running properly. You will also see that we are mounting the /etc/gluster.vol file to /gluster (remember to create this folder to mount to) rather than mounting a network path as you might when mounting an NFS share.

If you wanted you could also add more to your upstart script to handle clean un-mounting of gluster thus allowing you to then use the service gluster-mount (start|stop|restart) commands

A quick reboot of the client server should confirm that it boots successfully and you will now end up with your volume mounted at /gluster. You can now test this by creating a new file. I tend to create an empty file at /gluster/mounted just so I have a quick reference that the folder is mounted. Once that’s created if you now go and take a look at the /gluster/brick on your “peers” you should see that there is now a file called “mounted” sat there looking all smug that it worked.

Caveats

Some important things for you to be made aware of

Never write directly to a brick. Make sure to write to the volume only through a configured client
Beware of split-brain. http://community.gluster.org/q/what-is-split-brain-in-glusterfs-and-how-can-i-cause-it/ http://www.gluster.org/2012/06/healing-split-brain/
RTFM - Read The F***ing Manual. Gluster is big and complex and there is a lot for you to understand. You can download a copy of the manual from here

Our Redmine install died, We all cried!

Sat, 23 Feb 2013 00:00:00 +0000

We have been using redmine for quite a long time and a few months ago attempted to upgrade from 1.3 to 2.something. Unfortunately I (quite typically) borked the installation and since then its been hobbling along after my attempts to fix it left it crippled.

Yesterday it finally gave up the fight and my attempts to resurrect the installation were futile. After a quick funeral (the eulogy was very touching), and wake in a nearby emporium of alcoholic beverages to commiserate our loss, I set about trying to figure out what to do next.

Alternatives

Now while Redmine is a worthy tool and has always managed to do what I needed in the past, recently its just not cut the mustard. I’ve kept toying with the idea of creating our own project management system but as with all in-house projects that we dream up its just never going to happen.

A quick google around our options are to either go for a hosted solution (not possible as we have some very specific requirements regarding our SCM that mean we have to host our own repos for client work) or Redmine (or chilli project).

Yes we looked at a number of other management tools and of them all Redmine is still the closes to what we needed.

Installation

So I spin up a new server instance of ubuntu 12.10 on the cloud and get to work installing the latest version.

As root I then run through these steps (you should assume that ALL of these steps require you to be root and files should be owned by root)

# update/upgrade base installation of ubuntu packages
apt-get update && apt-get upgrade

# install the requisite scm tools that we use
apt-get install git-core subversion mercurial cvs

# set up ruby
apt-get install ruby rubygems libruby ruby-dev

# set up apache & mysql
apt-get install apache2 libapache2-mod-passenger mysql-server mysql-client libmysqlclient-dev

# install imagemagick and the magick wand
apt-get install imagemagick libmagickcore-dev libmagickwand5 libmagickwand-dev

# create our user and database in mysql 
# replace uniquePassword with your own password
mysql -u root -p -e "create user 'redmine'@'localhost' identified by 'uniquePassword'"
mysql -u root -p -e "create database redmine"
mysql -u root -p -e "grant all on redmine.* to 'redmine'@'localhost'"
mysql -u root -p -e "flush privileges"

# clone redmine code to target location
cd /usr/local/share
git clone git://github.com/redmine/redmine.git

# set apache as the owner of redmine
chown -R www-data:www-data redmine

# move into our new redmine folder
cd redmine

# set up your database configuration
cp config/database.yml.example config/database.yml
vim config/database.yml

production:
 adapter: mysql2
 database: redmine
 host: localhost
 username: redmine
 password: uniquePassword

# install bundler gem
gem install bundler

# use bundler to set up redmine installation and without specified dependencies
bundle install --without development test postgresql sqlite

# set up our secret token
rake generate_secret_token

# set up our database and load default configuration
RAILS_ENV=production rake db:migrate
RAILS_ENV=production rake redmine:load_default_data

# edit /etc/apache2/sites-available/default
<VirtualHost *:80>
 ServerAdmin webmaster@localhost
 ServerName mysite.co.uk
 ServerAlias www.mysite.co.uk
 DocumentRoot /usr/local/share/redmine/public
 <Directory />
 Options FollowSymLinks
 AllowOverride None
 </Directory>
 <Directory /usr/local/share/redmine/public>
 Options Indexes FollowSymLinks MultiViews
 AllowOverride All
 Order allow,deny
 allow from all
 </Directory>

 ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
 <Directory "/usr/lib/cgi-bin">
 AllowOverride None
 Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
 Order allow,deny
 Allow from all
 </Directory>

 ErrorLog ${APACHE_LOG_DIR}/error.log

 # Possible values include: debug, info, notice, warn, error, crit,
 # alert, emerg.
 LogLevel warn

 CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

# restart apache 
service apache2 restart

That should be enough for you to have a working installation of redmine ready for you to use/customise

Additional Config

We typically have additional steps that we would configure for our own installation.

# add plugin assets folder
mkdir /usr/local/share/redmine/public/plugin_assets
chown www-data:www-data /usr/local/share/redmine/public/plugin_assets

# enable some additional apache modules
a2enmod rewrite

# disable mod ssl
a2dismod ssl

# install gnutls 
apt-get install libapache2-mod-gnutls

# install ssl certificate bundle and key (this assumes that you have already copied the key and bundle to ~/)
mv ~/my_certificate.bnd /etc/ssl/certs/my_certificate.bnd
chmod 0644 /etc/ssl/certs/my_certificate.bnd
mv ~/my_certificate.crt /etc/ssl/private/my_certificate.key
chmod 0600 /etc/ssl/private/my_certificate.key

# now configure your /etc/apache2/sites-available/default-tls
<IfModule mod_gnutls.c>
<VirtualHost _default_:443>
 ServerAdmin webmaster@localhost
 ServerName mysite.co.uk
 ServerAlias www.mysite.co.uk
 DocumentRoot /usr/local/share/redmine/public
 <Directory />
 Options FollowSymLinks
 AllowOverride None
 </Directory>
 <Directory /usr/local/share/redmine/public>
 Options Indexes FollowSymLinks MultiViews
 AllowOverride All
 Order allow,deny
 allow from all
 </Directory>

 ErrorLog ${APACHE_LOG_DIR}/error.log
 # Possible values include: debug, info, notice, warn, error, crit, alert, emerg.
 LogLevel warn
 CustomLog ${APACHE_LOG_DIR}/ssl_access.log combined

 GnuTLSEnable On
 GnuTLSCertificateFile /etc/ssl/certs/my_certificate.bnd
 GnuTLSKeyFile /etc/ssl/private/my_certificate.key
 GnuTLSPriorities NORMAL:!DHE-RSA:!DHE-DSS:!AES-256-CBC:%COMPAT
</VirtualHost>
</IfModule>

# Add some Rails / Passenger specific config to /etc/apache2/sites-available/default-tls
RailsEnv production
PassengerDefaultUser www-data
PassengerSpawnMethod smart
PassengerPoolIdleTime 300
PassengerMaxRequests 5000
PassengerStatThrottleRate 5
PassengerHighPerformance On

# change your /etc/apache2/sites-available/default to redirect to ssl
<VirtualHost *:80>
 ServerAdmin sysadmin@zucchi.co.uk
 ServerName mysite.co.uk
 ServerAlias www.mysite.co.uk
 RewriteEngine On
 RewriteCond %{HTTPS} off
 RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
 Options FollowSymLinks
 AllowOverride None

 ErrorLog ${APACHE_LOG_DIR}/error.log
 LogLevel warn
 CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

# enable your new default-tls vhost and restart apache
a2ensite default-tls
service apache2 restart

# setup &amp; configure email
# when prompted select "internet site" and enter the domain you are hosting redmine from i.e. mysite.co.uk)
apt-get install postfix

# create config file and uncomment the production settings for sendmail
cp /usr/local/share/redmine/config/configuration.yml.example /usr/local/share/redmine/config/configuration.yml
vim /usr/local/share/redmine/config/configuration.yml

production:
 email_delivery:
 delivery_method: :sendmail

service apache2 restart

#install pixel cookers theme cos we like it
git clone git://github.com/pixel-cookers/RedmineThemePixelCookers.git /usr/local/share/redmine/public/themes/pixel-cookers

Quick and easy setup of and connection to NRPE on Ubuntu

Wed, 06 Feb 2013 00:00:00 +0000

About NRPE

NRPE (Nagios Remote Plugin Executor) is a useful tool that allows you to execute scripts on remote servers and return the output for ingestion by some form of monitoring software.

Setup

We currently have our own instance of Icinga running to monitor our servers and have recently started to offer access to it for our clients.

The majority of our servers (and our clients servers if we set them up) use one variant or another of Ubuntu. This means we can very quickly get our servers connected to a Nagios/Icinga instance.

First things first we need to install the nrpe server and all the associated plugins

apt-get install nagios-nrpe-server \
nagios-plugins-basic \
nagios-plugins \
nagios-plugins-extra

Next we need to edit the main nrpe config file to be found @ /etc/nagios/nrpe.cfg. What your looking for is the lines

# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames 
# that are allowed to talk to the NRPE daemon.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address. I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

allowed_hosts=127.0.0.1

# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments to commands that are executed. This option only works
# if the daemon was configured with the --enable-command-args configure script
# option. 
#
# *** ENABLING THIS OPTION IS A SECURITY RISK! *** 
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments

dont_blame_nrpe=0

You will want to change this to the IP of your Nagios/Icinga instance and set the dont_blame_nrpe value to 1. Feel free to take a look round the rest of the file. Its all quite interesting and generally will documented. Be careful what you change though in case something breaks.

You will also want to look for some lines that are refererd to as “COMMAND DEFINITIONS” and look something like this

command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib/nagios/plugins/check_load -w 15,10,5 -c 30,25,20
command[check_hda1]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/hda1
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200

You can go ahead and comment these out as we will be adding our own definitions shortly. The main reason for removing these is that we will be configuring some specific scripts for our own use later that allow you to configure your requirements and thereshold from within your Nagios/Icinga config.

Configuration of Monitoring Server

Once this is complete you can now configure a new “check command” for use with your nagios/icinga server.

define command {
 command_name check_nrpe
 command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

define command {
 command_name check_nrpe_command_args
 command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$ -a $ARG2$
}

Here you can see that we have set up 2 different check commands. The first is a simple command requiring only one argument of $ARG1$ which would be the name of the command we want to run on the remote server. The second command is almost identical except for the fact it takes a second argument which allows you to input a series of “arguments” to be passed to the command on your remote server. each argument should be separated by a space.

Now that you have these you can then configure your hosts and services to make use of it. I would recommend having a trawl through the Nagios/Icinga sites & documentation to find out how to create a config that suits you.

Configuration of Remote Server

Now that we have our monitoring server ready its time to add the command we want to run to the remote server.

To do this your /etc/nagios/nrpe.cfg shoudl hopefully have a line in it that looks like

include=/etc/nagios/nrpe_local.cfg

if it doesn’t have a line like that then add it and edit the `/etc/nagios/nrpe_local.cfg` file to look a little like this

command[check_apt]=/usr/lib/nagios/plugins/check_apt
command[check_users]=/usr/lib/nagios/plugins/check_users -w $ARG1$ -c $ARG2$
command[check_load]=/usr/lib/nagios/plugins/check_load -w $ARG1$ -c $ARG2$
command[check_disk]=/usr/lib/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -p /dev/sda1
command[check_procs]=/usr/lib/nagios/plugins/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200

These are a few simple commands that I tend to use most often. These translate to your “check_nrpe” commands like so

$ARG1$ = everything inside the square brackets [ ]
$ARG2$ = each of the $ARG?$ keys as a single string separated by a space

Once that’s done you should be able restart your nrpe server with `/etc/init.d/nagios-nrpe-server restart`

It really is that simple. Do bear in mind that because you can pass arbitrary arguments into nrpe this was you could leave yourself vulnerable to a bit of maliciousness so its a good idea to make sure your firewall restricts port 5666 (the default port) to IPs you trust.

Compiling Apache 2.4 on Ubuntu 12.04

Tue, 06 Nov 2012 00:00:00 +0000

I’ve decided that I need to up my game when it comes to webservers. However I’m not yet ready to switch to Nginx or one of the other webservers out in the wild as I need something up and running rapidly.

Granted the numbers are definitely against Apache in a lot of benchmarks but historically I’ve always had a good experience and the entry level makes it much more appropriate for me to stick with it.

However Apache 2.2 is rather long in the tooth, thankfully 2.4 has been out for a while now. The problem I have is that I tend to favour Ubuntu as a platform and there is no sign of a 2.4 version appearing on the horizon anytime soon as they are waiting for it to be implemented upsteam in Debian before including it in Ubuntu.

Now there are PPAs available out there but im not overly happy using them (especially on production environments) So the only option is to compile.

First thing is to install all the dependencies we are going to need. Thankfuly ubuntu has a nice and simple way of handling this.

apt-get build-dep apache2

We can then download the source code and start the compilation.

So from the root of our new copy of the source we need to run our configure.

./configure --prefix=/usr/local/apache2 \
 --enable-mods-shared=all \
 --enable-http \
 --enable-deflate \
 --enable-expires \
 --enable-slotmem-shm \
 --enable-headers \
 --enable-rewrite \
 --enable-proxy \
 --enable-proxy-balancer \
 --enable-proxy-http \
 --enable-proxy-fcgi \
 --enable-mime-magic \
 --enable-log-debug \
 --with-mpm=event

You will notice that I’m installing it using the event mpm. Hopefully I’ll be covering more about the event mpm in the future.

Next we need to run make

make && make install

Once that’s complete you should be able to run

/usr/local/apache2/bin/apachectl start

and get the “it works” message through your webrowser when accessing the server IP.

Dont forget to configure apache to suit your specific requirements.

Something that will come up is how to start apache on boot. Seeing as Ubuntu uses Upstart it makes sense to utilise it for controlling apache.

So in the file `/etc/ini/apache.conf` we need to put

# apache2 - http server
#
# Apache is a web server that responds to HTTP and HTTPS requests.
# Required-Start: $local_fs $remote_fs $network $syslog
# Required-Stop: $local_fs $remote_fs $network $syslog

author "Matt Cockayne <matt@zucchi.co.uk"
description "Apache 2.4 HTTP Server"

start on runlevel [2345]
stop on runlevel [!2345]

console output

pre-start script
 mkdir -p /var/run/apache2 || true
 install -d -o www-data /var/lock/apache2 || true
 # ssl_scache shouldn't be here if we're just starting up.
 # (this is bad if there are several apache2 instances running)
 rm -f /var/run/apache2/*ssl_scache* || true
end script

# Give up if restart occurs 10 times in 30 seconds.
respawn limit 10 30
respawn

script
 if test -f /usr/local/apache2/bin/envvars; then
 . /usr/local/apache2/bin/envvars
 fi
 ULIMIT_MAX_FILES="ulimit -S -n `ulimit -H -n`"
 if [ "x$ULIMIT_MAX_FILES" != "x" ] ; then
 $ULIMIT_MAX_FILES
 fi

 /usr/local/apache2/bin/httpd -k start -D FOREGROUND
end script

This is a rather simple upstart script and I will be looking to update it at some point… but it works

Once that’s done you should find that on reboot Apache will start and take advantage of all the management features of upstart including attempting to respawn Apache should it end unexpectedly. You should also be able to then use the following commands to control Apache.

# how to start start apache
start apache
# or 
initctl start apache

# how to stop apache
stop apache
# or 
initctl stop apache

# how to restart apache
restart apache 
# or 
initctl restart apache

# check the status of apache 
status apache
# or
initctl status apache

I generally tend to avoid using the apachectl script found at /usr/local/apache/bin/apachectl once upstart takes control.

Compiling PHP 5.4 on Ubuntu 12.04

Tue, 06 Nov 2012 00:00:00 +0000

So recently I’ve been working with PHP 5.4 a LOT. Unfortunately Ubuntu (my main dev environment) is behind the times. So I’m resorting to compiling PHP manually.

Not a daunting as it may first appear. The really tricky part is working out your dependencies and `configure` script.

Hence the reason for this post as a reminder for myself and others that may want to do a quick compile. (I would recommend that if your compiling for a production/live environment that you make sure you understand what it is your compiling though before just using what’s here)

So where to start. Dependencies first I think

Ubuntu allows you to install dependencies for building source `apt-get build-deps`. We will use this and install any extras we may need.

apt-get install \
libxml2 \
libxml2-dev \
libssl-dev \
pkg-config \
curl \
libcurl4-nss-dev \
enchant \
libenchant-dev \
libjpeg8 \
libjpeg8-dev \
libpng12-0 \
libpng12-dev \
libvpx1 \
libvpx-dev \
libfreetype6 \
libfreetype6-dev \
libt1-5 \
libt1-dev \
libgmp10 \
libgmp-dev \
libicu48 \
libicu-dev \
mcrypt \
libmcrypt4 \
libmcrypt-dev \
libpspell-dev \
libedit2 \
libedit-dev \
libsnmp15 \
libsnmp-dev \
libxslt1.1 \
libxslt1-dev

And now the configure

./configure \
--prefix=/usr/local/php \
--with-apxs2=/usr/local/apache2/bin/apxs \
--enable-fpm \
--with-fpm-user=www-data \
--with-fpm-group=www-data \
--with-config-file-path=/usr/local/php/conf \
--with-config-file-scan-dir=/usr/local/php/conf.d \
--enable-debug \
--with-openssl \
--with-kerberos \
--with-zlib \
--enable-calendar \
--with-curl \
--with-curlwrappers \
--with-enchant \
--enable-exif \
--enable-ftp \
--with-gd \
--with-jpeg-dir=/usr \
--with-png-dir=/usr \
--with-vpx-dir=/usr \
--with-freetype-dir=/usr \
--with-t1lib \
--enable-exif \
--enable-gd-native-ttf \
--enable-gd-jis-conv \
--with-gettext \
--with-gmp \
--with-mhash \
--enable-intl \
--enable-mbstring \
--with-mcrypt \
--with-mysql \
--with-mysqli \
--enable-pcntl \
--with-pdo-mysql \
--with-pdo-pgsql \
--with-pgsql \
--with-pspell \
--with-libedit \
--with-readline \
--enable-shmop \
--with-snmp \
--enable-soap \
--enable-sockets \
--enable-sysvmsg \
--enable-sysvshm \
--with-xsl \
--enable-zip \
--with-pear \
--enable-zend-signals \
--enable-maintainer-zts

Once these are done then we follow the standard make process. Notice we are also running make test… very important as it givges more data for the developers to work with.

make && make test && make install

The next thing is configuring your php.ini file as the install doesn’t have one yet so we copy either the production or development default from the source code to the new conf dir and edit to suit your needs.

cp {php-source-dir}/php.ini-(development|production) /usr/local/php/conf

Thats it. All ready to roll… almost, this installation is the one I use for use with a webserver so you will want to add the appropriate directives to apache.

LoadModule php5_module modules/libphp5.so

AddHandler php5-script .php
AddType text/html .php

Rsync and custom SSH commands

Tue, 31 Jul 2012 00:00:00 +0000

Rsync is a great tool but can be a pain if you have to jump through hoops to connect via ssh such as connecting via a different port.

A simple solution is to use the -e flag (also knows as –rsh=COMMAND). This flag allows you manually define the ssh command to use when connecting

rsync -e 'ssh -p2020' -rav ./* user@server:

Will allow me to connect to a server with SSH listening on port 2020

Loaded Testing

Sat, 30 Jun 2012 00:00:00 +0000

I recently had to do some load testing for a site recently that would allow me to test in excess of 100k requests in a 60 second period…

So I decided to do some testing using JMeter as it seemed like a suitable tool for doing what I needed and I had used it for some simpler testing in the past.

After a little fumbling around I managed to get a test plan designed that would simulate 10k users actually navigating the site and adding to a cart etc, with a number of various interactions. It wasnt perfect but it would correctly simulate over 100k requests.

So feeling quite pleased with myself I started the test from my laptop. Now I’m not a big gamer, I’m known to play a little World or Warcraft from time to time but that’s about it. So when it comes to computing power i tend to opt for battery life over sheer grunt.

Suffice to say, my laptop fell flat on its face, and if it hadn’t it turns out that the connection I was using just wasn’t up to the task of handling that much traffic adequately.

So plan B…

I quickly fired up the largest AWS instance available and got a copy of jmeter installed. A little tinkering with my test plan and some googling on how to run jmeter without a gui and a quick

./jmeter -n -t test-plan.jmx

and it appeared to be running.

(Please bear in mind that I’m being overly kind… it took a LOT of tinkering and twice as much Googling to work out how to get the test results out so i could actually get some idea of WTF was happening during the test)

So… client “happy”… I decided to go and find a better way to do my load testing in the future.

Sticking with JMeter I managed to find this gem of a page

http://jmeter.apache.org/usermanual/remote-test.html

tl;dr > use your local install of jmeter to trigger tests to run on one or more remote “nodes” and then have all the results sent to your local install.

So I set to work!

Building a Node

First I need to set up an AWS instance that we can use and duplicate so I can quickly build a cluster of nodes on demand. I’m a big fan of Ubuntu so I spin up a micro instance of 12.04 server. Next I shell into the instance and install the default Java runtime from apt

apt-get install openjdk-7-jre

Yes I know there are other more appropriate runtimes, but i dont really care… i just need it to work and it does.

next I grab a copy of the latest stable from http://jmeter.apache.org/download_jmeter.cgi and un-tar it to /usr/local/jmeter

(N.B. JMeter is available through the apt but I had issues with that version and you need to make sure that both your local version and all the nodes run the same version of jmeter)

We can now test that the install is working running /usr/local/jmeter/bin/jmeter-server and you should get some output that looks similar to

Created remote object: UnicastServerRef [liveRef: [endpoint:[10.???.???.???:38939](local),objID:[46522b57:138381f1023:-7fff, 2635011707874933136]]]

Which tells us that the server is running.

BUT unfortunately its not going to work just yet. Because we are using Amazons EC2 we are going to relying on their NAT for routing. Out of the box JMeter just wont work properly.

However there is something we can do to combat this. We can set the parameter RMI_HOST_DEF that the /usr/local/jmeter/bin/jmeter-server script will include in starting the server.

export RMI_HOST_DEF=-Djava.rmi.server.hostname=$(wget http://169.254.169.254/latest/meta-data/public-hostname -q -O -)

I’ll explain what we are doing here. Amazon have been quite clever by providing a meta-data endpoint that you can poll from within your instance to get key pieces of data… Including the public dns record.

We can use this endpoint and using wget pipe that into the RMI_HOST_DEF param (ensuring that we prepend -D) and then export that so it becomes available to the /usr/local/jmeter/bin/jmeter-server script.

Now to get the server to start on boot.

a quick upstart script should solve this

# Upstart script to initialise jmeter-server

description "JMeter Server"
author "Dev in Charge "

start on started networking
stop on stopping networking
stop on stopping shutdown

console output

script
 # get the current public DNS record
 export RMI_HOST_DEF=-Djava.rmi.server.hostname=$(wget http://169.254.169.254/latest/meta-data/public-hostname -q -O -)

 # start jmeter in server mde
 /usr/local/jmeter/bin/jmeter-server
end script

saving this to /etc/init/jmeter-server.conf will mean that it will auto-start jmeter-server on boot and allow you to manually control the process using start jmeter-server and stop jmeter-server

and thats it… instance configured

All you need to do now is save the instance as an AMI and you have an on-demand image for spinning up a cluster of remote JMeter servers for you to play with.

Configuring your local installation

Now that the server side is working we need to configure our local installation to allow it to connect.

First things first however, make sure you are using the same version of JMeter as you are running on the server.

We need to edit the jmeter.properties file that can be found in the bin folder of the installtion you downloaded. Look for the parameter remote_hosts This needs to be set with the public dns of the remote server(s) your connecting to. for example

remote_hosts=ec2-176-34-164-170.eu-west-1.compute.amazonaws.com,ec2-123-34-456-789.eu-west-1.compute.amazonaws.com

Thats your local version configured. You will now be able to tell your local version to run tests on any or all of your specified remotes.

However if your like me you work behind a router/firewall. If so this isnt the end of the story. When you send a test plan to a remote from your local install it will also send the IP address of your local machine for it to send the results back to. JMeter does this by looking up where your current hostname resolves to. In my circumstance it resolved to 127.0.1.1. The reason it did this is down to the fact my systems host file had the line

127.0.1.1 devincharge.local

To resolve this I had to change it to my external IP address

89.345.871.79 devincharge.local

And set up port forwarding from my router to my local machine for all ports from 1024 to 65535. Now, you can if you want use specific ports so you dont have to port forward everything from your router, but i’ll leave that for you to lookup as there are plenty resources on how to do this for you to google and I’ve waffled on for far too long already.

Happy testing