<!DOCTYPE linuxdoc system[ <!ENTITY CurrentVer "0.90.2 - Alpha">  <!ENTITY mdstat "<TT>/proc/mdstat</TT>">  <!ENTITY ftpkernel "<TT>ftp://ftp.fi.kernel.org/pub/linux</TT>">  <!ENTITY fstab "<TT>/etc/fstab</TT>">  <!ENTITY raidtab "<TT>/etc/raidtab</TT>">]><ARTICLE><TITLE>The Software-RAID HOWTO<AUTHOR>Jakob &Oslash;stergaard        (<htmlurl        url="mailto:jakob@ostenfeld.dk"        name="jakob@ostenfeld.dk">)<DATE>v. &CurrentVer;, 27th february 1999<ABSTRACT>This HOWTO describes how to use Software RAID under Linux. You must beusing the RAID patches available from <htmlurlname="ftp://ftp.fi.kernel.org/pub/linux/daemons/raid/alpha"url="ftp://ftp.fi.kernel.org/pub/linux/daemons/raid/alpha">. The HOWTOcan be found at <htmlurlname="http://ostenfeld.dk/~jakob/Software-RAID.HOWTO/"url="http://ostenfeld.dk/~jakob/Software-RAID.HOWTO/">.</ABSTRACT><TOC><SECT>Introduction<P>This howto is written by Jakob &Oslash;stergaard based on a largenumber of emails between the author and Ingo Molnar (<htmlurlurl="mailto:mingo@chiara.csoma.elte.hu"name="mingo@chiara.csoma.elte.hu">) -- one of the RAID developers --,the linux-raid mailing list (<htmlurlurl="mailto:linux-raid@vger.rutgers.edu"name="linux-raid@vger.rutgers.edu">) and various other people.<P>The reason this HOWTO was written even though a Software-RAID HOWTOallready exists is, that the old HOWTO describes the old-styleSoftware RAID found in the stock kernels. This HOWTO describes the useof the ``new-style'' RAID that has been developed more recently. Thenew-style RAID has a lot of features not present in old-style RAID.<P>Some of the information in this HOWTO may seem trivial, if you knowRAID all ready. Just skip those parts.<P><SECT1>Disclaimer<P>The mandatory disclaimer:<P>Although RAID seems stable for me, and stable for many other people,it may not work for you.  If you loose all your data, your job, gethit by a truck, whatever, it's not my fault, nor the developers'.  Beaware, that you use the RAID software and this information atyour own risk!  There is no guarantee whatsoever, that any of thesoftware, or this information, is in anyway correct, nor suited forany use whatsoever. Back up all your data before experimenting withthis. Better safe than sorry.<P><SECT1>Requirements<P>This HOWTO assumes you are using a late 2.2.x or 2.0.x kernel with amatching raid0145 patch, and the 0.90 version of the raidtools. Bothcan be found at <HTMLURLname="ftp://ftp.fi.kernel.org/pub/linux/daemons/raid/alpha"url="ftp://ftp.fi.kernel.org/pub/linux/daemons/raid/alpha">. The RAIDpatch, the raidtools package, and the kernel should all match as closeas possible. At times it can be necessary to use older kernels if raidpatches are not available for the latest kernel.<P><SECT>Why RAID ?<P>There can be many good reasons for using RAID. A few are; the abilityto combine several physical disks into one larger ``virtual'' device,performance improvements, and redundancy.<P><SECT1>Technicalities<P>Linux RAID can work on most block devices. It doesn't matter whetheryou use IDE or SCSI devices, or a mixture. Some people have also usedthe Network Block Device (NBD) with more or less success.<P>Be sure that the bus(ses) to the drives are fast enough. You shouldn'thave 14 UW-SCSI drives on one UW bus, if each drive can give 10 MB/sand the bus can only sustain 40 MB/s.Also, you should only have one device per IDE bus. Running disks asmaster/slave is horrible for performance. IDE is really bad ataccessing more that one drive per bus.  Of Course, all newermotherboards have two IDE busses, so you can set up two disks inRAID without buying more controllers.<P>The RAID layer has absolutely nothing to do with the filesystemlayer. You can put any filesystem on a RAID device, just like anyother block device.<P><SECT1>Terms<P>The word ``RAID'' means ``Linux Software RAID''. This HOWTO does nottreat any aspects of Hardware RAID.<P>When describing setups, it is useful to refer to the number of disksand their sizes. At all times the letter <BF>N</BF> is used to denotethe number of active disks in the array (not countingspare-disks). The letter <BF>S</BF> is the size of the smallest drivein the array, unless otherwise mentioned. The letter <BF>P</BF> isused as the performance of one disk in the array, in MB/s. When used,we assume that the disks are equally fast, which may not always be true.<P>Note that the words ``device'' and ``disk'' are supposed to mean aboutthe same thing.  Usually the devices that are used to build a RAIDdevice are partitions on disks, not necessarily entire disks.  Butcombining several partitions on one disk usually does not make sense,so the words devices and disks just mean ``partitions on differentdisks''.<P><SECT1>The RAID levels<P>Here's a short description of what is supported in the Linux RAIDpatches. Some of this information is absolutely basic RAID info, butI've added a few notices about what's special in the Linuximplementation of the levels.  Just skip this section if you knowRAID. Then come back when you are having problems   :)<P>The current RAID patches for Linux supports the followinglevels:<ITEMIZE><ITEM><BF>Linear mode</BF><ITEMIZE><ITEM>Two or more disks are combined into one physical device. The disksare ``appended'' to each other, so writing to the RAID device will fillup disk 0 first, then disk 1 and so on. The disks does not have to beof the same size. In fact, size doesn't matter at all here   :)<ITEM>There is no redundancy in this level. If one disk crashes you willmost probably loose all your data.  You can however be lucky torecover some data, since the filesystem will just be missing one largeconsecutive chunk of data.<ITEM>The read and write performance will not increase for singlereads/writes. But if several users use the device, you may be luckythat one user effectively is using the first disk, and the other useris accessing files which happen to reside on the second disk. If thathappens, you will see a performance gain.</ITEMIZE><ITEM><BF>RAID-0</BF><ITEMIZE><ITEM>Also called ``stripe'' mode. Like linear mode, except that reads andwrites are done in parallel to the devices. The devices should haveapproximately the same size. Since all access is done in parallel, thedevices fill up equally. If one device is much larger than the otherdevices, that extra space is still utilized in the RAID device, butyou will be accessing this larger disk alone, during writes in thehigh end of your RAID device. This of course hurts performance.<ITEM>Like linear, there's no redundancy in this level either. Unlikelinear mode, you will not be able to rescue any data if a drivefails. If you remove a drive from a RAID-0 set, the RAID device willnot just miss one consecutive block of data, it will be filled withsmall holes all over the device. e2fsck will probably not be able torecover much from such a device.<ITEM>The read and write performance will increase, because reads andwrites are done in parallel on the devices. This is usually the mainreason for running RAID-0. If the busses to the disks are fast enough,you can get very close to N*P MB/sec.</ITEMIZE><ITEM><BF>RAID-1</BF><ITEMIZE><ITEM>This is the first mode which actually has redundancy. RAID-1 can beused on two or more disks with zero or more spare-disks. This mode maintainsan exact mirror of the information on one disk on the otherdisk(s). Of Course, the disks must be of equal size. If one disk islarger than another, your RAID device will be the size of thesmallest disk.<ITEM>If up to N-1 disks are removed (or crashes), all data are still intact. Ifthere are spare disks available, and if the system (eg. SCSI driversor IDE chipset etc.) survived the crash, reconstruction of the mirrorwill immediately begin on one of the spare disks, after detection ofthe drive fault.<ITEM>Read performance will usually scale close to to N*P, while write performance isthe same as on one device, or perhaps even less.  Reads can be done inparallel, but when writing, the CPU must transfer N times as much datato the disks as it usually would (remember, N identical copies ofall data must be sent to the disks).</ITEMIZE><ITEM><BF>RAID-4</BF><ITEMIZE><ITEM>This RAID level is not used very often. It can be used on threeor more disks. Instead of completely mirroring the information, itkeeps parity information on one drive, and writes data to the otherdisks in a RAID-0 like way.  Because one disks is reserved for parityinformation, the size of the array will be (N-1)*S, where S is thesize of the smallest drive in the array. As in RAID-1, the disks should eitherbe of equal size, or you will just have to accept that the S in the(N-1)*S formula above will be the size of the smallest drive in thearray.<ITEM>If one drive fails, the parityinformation can be used to reconstruct all data.  If two drives fail,all data is lost.<ITEM>The reason this level is not more frequently used, is becausethe parity information is kept on one drive. This information must beupdated <EM>every</EM> time one of the other disks are writteto. Thus, the parity disk will become a bottleneck, if it is not a lotfaster than the other disks.  However, if you just happen to have alot of slow disks and a very fast one, this RAID level can be very useful.</ITEMIZE><ITEM><BF>RAID-5</BF><ITEMIZE><ITEM>This is perhaps the most useful RAID mode when one wishes to combinea larger number of physical disks, and still maintain someredundancy. RAID-5 can be used on three or more disks, with zero ormore spare-disks. The resulting RAID-5 device size will be (N-1)*S,just like RAID-4. The big difference between RAID-5 and -4 is, thatthe parity information is distributed evenly among the participatingdrives, avoiding the bottleneck problem in RAID-4.<ITEM>If one of the disks fail, all data are still intact, thanks to theparity information. If spare disks are available, reconstruction willbegin immediately after the device failure.  If two disks failsimultaneously, all data are lost. RAID-5 can survive one diskfailure, but not two or more.<ITEM>Both read and write performance usually increase, but it's hard topredict how much.</ITEMIZE></ITEMIZE><SECT2>Spare disks<P>Spare disks are disks that do not take part in the RAID set until oneof the active disks fail.  When a device failure is detected, thatdevice is marked as ``bad'' and reconstruction is immediately startedon the first spare-disk available.<P>Thus, spare disks add a nice extra safety to especially RAID-5 systemsthat perhaps are hard to get to (physically). One can allow the systemto run for some time, with a faulty device, since all redundancy ispreserved by means of the spare disk.<P>You cannot be sure that your system will survive a disk crash. TheRAID layer should handle device failures just fine, but SCSI driverscould be broken on error handling, or the IDE chipset could lock up,or a lot of other things could happen.<P><SECT1>Swapping on RAID<P>There's no reason to use RAID for swap performance reasons. The kernelitself can stripe swapping on several devices, if you just give themthe same priority in the fstab file.<P>A nice fstab looks like:<VERB>/dev/sda2       swap           swap    defaults,pri=1   0 0/dev/sdb2       swap           swap    defaults,pri=1   0 0/dev/sdc2       swap           swap    defaults,pri=1   0 0/dev/sdd2       swap           swap    defaults,pri=1   0 0/dev/sde2       swap           swap    defaults,pri=1   0 0/dev/sdf2       swap           swap    defaults,pri=1   0 0/dev/sdg2       swap           swap    defaults,pri=1   0 0</VERB>This setup lets the machine swap in parallel on seven SCSI devices. Noneed for RAID, since this has been a kernel feature for a long time.<P>Another reason to use RAID for swap is high availability.  If you setup a system to boot on eg. a RAID-1 device, the system should be ableto survive a disk crash. But if the system has been swapping on thenow faulty device, you will for sure be going down.  Swapping on theRAID-1 device would solve this problem. <P>However, swap on RAID-{1,4,5} is <BF>NOT</BF> supported. You can set it up,but it will crash. The reason is, that the RAID layer sometimesallocates memory before doing a write. This leads to a deadlock, sincethe kernel will have to allocate memory before it can swap, and swapbefore it can allocate memory.<P>It's sad but true, at least for now.