Skip to content

The little Raspberrypi computer, designed to re-create that wonderful sense of being able to do things yourself that those of us who cut our teeth in the early days of personal computing thrived on, is bringing the fun of general purpose computing back to all ages, especially kids.  It's wonderful seeing the blog posts and Twitter comments from parents especially, proud of the achievements of even very young children, on this capable little machine.  The combination of that sense of the possible along with an astonishing price tag of around £25 has resulted in great success, well worthy of the foresight of the small and principled team that made it possible.

But can it run a DSpace archive?

And the answer is Yes!  Well, maybe yeeees-ish...

While in general, Free Software tends to be pretty efficient in terms of its resource use, java-based software like DSpace isn't known for this aspect of its operation.  But here we are, the Pi in operation, dwarfed by the only USB network stick for network access I had handy:-

RaspberryPi running DSpace

The drawback isn't memory, of which the Pi only has 256MB (although the latest versions have - phoooaaarr - 512MB) but rather the hit on the  processor.  Start-up takes over 10 minutes, and the first time each page is displayed the processor works hard.  Below is a performance monitor screen shot and you can see the processes and the flat-out working of the processor:-

I know I can, I think I can...

And while it doesn't show much, here's the DSpace instance freshly minted:-

Squeaky new DSpace screen

The first time you click on a new screen, it takes a while to get there, but after the first time, the speed is quite reasonable.

So yes, it can be done, and it's been an interesting exercise.  I used the standard Raspian Linux distribution for this, and it was very straight forward to do the installation.

Why would you do this?  Well in the Raspberrypi tradition, because you can.  But there may be other reasons too.  It may be useful in some instances to have a heavily distributed  series of DSpace instances each of which gets a light impact but which together form a significant archive.  It may be that a particular project needs an Archive but has an impossibly tight budget, but that needs to stick to standard software, and with the Pi, you'll get change from £50 after buying a wireless USB stick and SD-card storage.  Or, and this is an interesting use case, you may need an Archive somewhere where the power supply is very limited.  The Pi needs less than 5 watts to run.

As part of the winter learning programme developed by Assynt Learning, training in the use of the Archive will be delivered in February and March 2013.  It is a sign of the community nature of the Archive that Assynt Learning's welcome involvement is taking place.  Anyone interested in taking part should contact Sharon or  Sandra at the Leisure Centre on 844123

The Assynt Community Digital Archive is built mostly on five virtual servers, separating services across virtual machines to aid resilience and ease system updates by containing risk to just the service in question.  This tactic has other advantages, as we shall see.  Four of the virtual servers are required, and the fifth is a remote access virtual desktops system, not strictly required, but definitely useful, for a disaster recovery situation.  This article uses technical terms which may be most meaningful to a technical reader but the outcome should be of general interest.

Just before Christmas 2012, remote monitoring software showed that the entire archive was down, all five virtual machines, but the firewall, a separate little unit, was still accessible via ssh, and through the firewall access was possible to a tiny £100 test server Stevan uses (an HP microserver with just 1GB of memory and a 2TB disk.)  The test server is also used as a staging post for backups, the main systems backing up to the small server, and from there, copied to removable disks for offsite storage.  As it happened, Stevan was unable to go to the Archive for a few weeks, but this represented a good opportunity to look at disaster recovery.  One of the glories of Free and Open Source software is that it tends to be quite modest in terms of hardware requirements.  With virtual systems, you can also cut your cloth to suit, as will be shown.

The backups already on the little microserver already contained all the data, but, as it happened,  copies had been taken of the main virtual server containers just a few weeks before.  As a test server, it was already set up as a KVM virtual system host. It was therefore simple to copy over the various XML definition files for each virtual system.  This meant modifying each file slightly, as there were differences between the file format on the Debian Squeeze installation on the microserver, and the Scientific Linux installation on the main server.  (This difference is now a thing of the past, as the main server now also runs Debian.)  Other edits were to change the location of the container files, though it may  have been possible to put in a symlink to the location as well.  However, edits were additionally required to make sure the memory requirements for each server fitted into the miserably small 1GB of memory available.  This was done without much trouble, and each server was brought up with manual commands.  This is quite important, as automatic starts are not wanted on this server.  If it rebooted for some reason, and a second instance of the same virtual machine tried to come up on the network , address clashing would occur.

The commands are quite straight forward - "virsh create <XML Definition file>" to set up the virtual machine followed by "virsh start <MACHINE_NAME>" to start it.

So within an hour or two the basic system was back up and running.  We were a tiny bit fortunate in that no work at all had been done on the Archive itself since the last backup, so it was possible to restore the systems to the last backup point quite quickly.  What is more, the entire process was done remotely, in this case from just 7 miles away, but it would not have mattered had the systems been on the the side of the planet.

Later, Stevan was able to go in to the Archive, the additional flexibility of virtual system then came to the fore.  A small desktop machine was brought into service running Debian Squeeze, and some of the virtual machines were transferred to this machine, reducing the load on the brave little microserver.

This was obviously a full DR exercise rather than a mere test, and worked well.  Some learnings to ease the process have been noted for future reference, but in these instances, it is often less of a case of blindly following a disaster recovery plan and more of a case of having the required flexibility to work with what one has.  A virtualised Free Software environment certainly provides that.  While a community archive suffers merely inconvenience rather than business or financial peril in the event of a complete disaster it is still a source of professional pride to be able to deliver such a robust and resilient solution.

The cause of the main server failure, as noted in the title, was the failure of three of the four disks in the IBM RAID array.  All three (bit not the fourth) had the same manufacturing date.  All three were Seagates, which, had they been bought from Seagate, would have had a five year warranty, but IBM only offers a year on the exact same disks.  Details about this event are included here

 

At the start of the Assynt Community Digital Archive project, there were a lot of unknowns.  Among these were the likely initial disk capacity requirements and how to implement them.  It quickly became clear that the Archive would require more than just a machine in the corner, as it had to run a fullish range of network services to be long lasting.  Among these were directory services, email, deployment (web) services, security services and remote access capabilities.  These would be deployed as virtualised services with considerable separation of services, rather than simply running a single server, which would have been possible.  The main reason for this was resilience and the opportunity to make changes to one sub-system without affecting the others, a principle which has worked well for me in the past.  It's the type of design decision that only experience of what it takes to run services after the initial installation can bring.

The possibility of greater complexity than is ideal therefore raised its head, and that affected the choice of hardware.  Rather than go for a NAS storage option, the choice was made for an initial local storage RAID system on the main server.  This was purely an attempt to lessen the number of systems running on the network, and therefore an attempt to reduce complexity.  The cost differences were not that great.  The choice of hardware RAID, though, was a bad one.

The server itself had to be a named brand, and as I had had good success with IBM's x86 server range, we plumped for an x3200 running SATA disks on IBM's M1015 RAID card, an OEM version of an LSI RAID controller.  This was the first time I had used a SATA RAID controller, previous experience being SCSI based.

The first problem was the Debian Squeeze, in 2010, did not support the controller as part of a standard installation.  For reasons stated above, I did not want a special arrangement to shoe-horn Squeeze onto the machine, but Red Hat based distributions worked well enough.  While previously CentOS was the red Hat clone to go for, at that time, a lack of resources had left them without updates for quite a while, and their future was unclear.  Scientific Linux, though, the Red Hat clone developed and supported by CERN, Fermi Labs and others, looked as though it was a god option, and installation was a breeze.  All, that is, except for one compromise, which was that Red Hat and their clones do not support JFS as standard.  We'll leave the file system choice for another post.

Having installed the base system as a KVM virtual machine host, performance testing of the disks threw up horrors.  Write speeds were pitiful, and every now and again the whole machine would simply stop responding for minutes at a time.  To cut a long googling story short, the issue is that the basic RAID card does not have a battery back-up for an onboard cache, and also no cache, and the effect on write speeds in particular is quite astonishing.  It's hard to give an estimate of the speeds at that time, because it varied so wildly, but initial system caching was lightning fast, while subsequent speeds dropped to zero at times.

Then the disks started failing.  Again, let's cut the long story short.  Three disk, all Seagates with the same manufacturing date, failed.  IBM only support SATA disks for a year, though the Seagate version of the same OEM disks are warrantied for 5 years.  These two events, the pitiful performance and the disk failure, leave a string impression and have dented my confidence in lower end IBM kit. By the time the server went dead, when the disks finally failed lemming-like, I had reason to bless the decision to implement as virtual machines, and the system could be transferred to desktop machines running a KVM host within a  few hours.  Again, the recovery process deserves its own post.

The kitty was also bare, as is to be expected with community projects which do not generate an income, so the only thing to be done was to be creative.  This entailed sacrificing four of the external removable backup disks, which contained 2TB Western Digital Caviar disks, and cannibalising them for the IBM.

By this time (January 2013) Debian Wheezy was maturing, and as Debian is used on all the VMs, it would be nicer if Debian was also running on the bare metal.  This time, Debian detected the RAID controller, allowing implementation of JFS.  Wheezy also recognised an EFI based system and installed itself accordingly.  The RAID was created as three virtual disks, one small one for the Debian plus KVM system, one 1TB partition for the virtual machine containers and one 2.6TB partition for the actual archive data.

Performance under Wheezy and JFS was much more consistent, but still absolutely dreadful.  I cannot believe IBM sell or sold the M1015 as a RAID5 controller without the additional hardware that apparently allows it to perform.  On testing, though, if the JFS partitions are mounted with the "sync" option, the performance stops fluctuating wildly, once the system resources are used up, BUT performance then peaks at around 4.5MB/s a good 10 times slower than it ought to be.  It is therefore a toss-up as to whether or not to allow cached performance for the usual smaller writes, while stunning the machine when larger writes are necessary, or to favour consistent operation over occasional flashes of acceptable speed.

The message is simple - don't buy an M1015 if you want to write to the disks at anything other than glacial pace.

It has to be said that, in practice, this isn't a killer issue, but more of an occasional hassle.  However, it does feel as though it is something hanging over one's head, and is an unnecessary distraction.

The one alternative, which is attractive, is to use software RAID.  But that then defeats the point of having a nice managed RAID system, and when the disks couldn't be trusted, sounded like a daft idea.  It is also necessary to boot from a separate disk, of course, further negating redundancy.  The portability  of the overall system may allow testing this at some point in the future though.

It seems that many communities have a need to archive information or to maintain a local store of heritage and culture.  In Assynt, in the north west Highlands of  Scotland, the community had exactly that urge, but the physical space and continuous commitment required for a physical archive seemed insurmountable.  A digital archive was therefore proposed and has been implemented.  This blog notes the various aspects of choices made, technical issues encountered, and other design choices associated with community archiving.  It is hoped that this information will be useful to others.