How to share the NDAS disk with OCFS2 file system
OCFS2 is a shared symmetric file system by Oracle. Unlikely OCFS, OCFS2 is for general purpose and included in 2.6.x offcial kernel tree.
We did test with Ubuntu system to use OCFS2.
- One NDAS disk.
- Two systems with i386 Ubuntu 5.10 installed.
Any system that has OCFS2 kernel module/feature installed.
Install NDAS driver for each system.
Make sure the NDAS disk is accessible.
Install ocfs tools on both system
apt-get install ocfs2-tools ocfs2console
Configure cluster /etc/ocfs2/cluster.conf on both system
node: ip_port = 7777 ip_address = 192.168.0.100 number = 1 name = host01 cluster = ocfs2 node: ip_port = 7777 ip_address = 192.168.0.101 number = 0 name = host02 cluster = ocfs2 cluster: node_count = 2 name = ocfs2
- The file format is very strict, indent with TABs. Even then I couldn't get it to work and had to resort to ocfs2console
don't forget to add the following to /etc/hosts on both system
host01 192.168.0.100 host02 192.168.0.101
Make sure port 7777 is reachable from each other (check iptables)
Start OCFS2 service on both system
/etc/init.d/o2fs load # depending on your system, this can also be o2cb instead of o2fs /etc/init.d/o2fs online # can also be o2cb
Partition the NDAS disk and format as ocfs2 on one system.
fdisk /dev/ndas-00xxxxxx:0 # path will be given by ndasadmin enable mkfs -t ocfs2 /dev/ndas-00xxxxxx:0p1 # p1 means the first partition
Turn on the heat beat for the file system on both system
ocfs2_hb_ctl -S -d /dev/ndas-00xxxxxx:0p1
Mount the file system from both systems
mkdir -p /mnt/ocfs2 mount -t ocfs2 /dev/ndas-00xxxxxx:0p1 /mnt/ocfs2
The ocfs2 nodes exchanges the heartbeats each other. During the file transfer the heartbeats could be congested by the NDAS i/o transfer. This congenstion results the 40% degration of writing performance.
Therefore, we recommend the hearbeat network should be dedicated to avoid NDAS i/o tranfer congestion. or use gigibit switch/nic if your NDAS disk is 100Mbps version. (With Gigabit NDAS disk, only solution is dedicating the heartbeat network.)
512MiB DRAM 1GiB transfer, HDD cache 2MB(affecting write performance)
NDAS/ocfs2/100Mbps/heartbeat on: read 9.7 MB/s , write 6.2 MB/s NDAS/ocfs2/100Mbps/heartbeat off: read 9.9 MB/s , write 10.4 MB/s NDAS/ext2/100Mbps: read 10.1 MB/s , write 10.8 MB/s Local HDD/ocfs2: read 39.8 MB/s, write 42.9 MB/s Local HDD/ext2Ï: read 44.7 MB/s , write 42.9 MB/s
- Please note that the hearbeat network is not dedicated. that's why the write of NDAS/ocfs2 is much slower (6.2 MB/s), I believe.
The network maxium speed of 100Mbps network is about 12.5 MB/s in theory if we ignore the overhead of packet headers metadata. so bottleneck for network test is the network speed, for local hdd is the hdd speed.