]>
Commit | Line | Data |
---|---|---|
7c673cae FG |
1 | |
2 | Last updated: 2017-04-08 | |
3 | ||
4 | The FreeBSD build will build most of the tools in Ceph. | |
11fdf7f2 | 5 | Note that the (kernel) RBD dependent items will not work |
7c673cae FG |
6 | |
7 | I started looking into Ceph, because the HAST solution with CARP and | |
8 | ggate did not really do what I was looking for. But I'm aiming for | |
9 | running a Ceph storage cluster on storage nodes that are running ZFS. | |
10 | In the end the cluster would be running bhyve on RBD disk that are stored in | |
11 | Ceph. | |
12 | ||
13 | The FreeBSD build will build most of the tools in Ceph. | |
14 | ||
15 | Progress from last report: | |
16 | ========================== | |
17 | ||
18 | Most important change: | |
19 | - A port is submitted: net/ceph-devel. | |
20 | ||
21 | Other improvements: | |
22 | ||
23 | * A new ceph-devel update will be submitted in April | |
24 | ||
25 | - Ceph-Fuse works, allowing to mount a CephFS on a FreeBSD system and do | |
26 | some work on it. | |
27 | - Ceph-disk prepare and activate work for FileStore on ZFS, allowing | |
28 | easy creation of OSDs. | |
29 | - RBD is actually buildable and can be used to manage RADOS BLOCK | |
30 | DEVICEs. | |
11fdf7f2 | 31 | - Most of the awkward dependencies on Linux-isms are deleted only |
7c673cae FG |
32 | /bin/bash is there to stay. |
33 | ||
34 | Getting the FreeBSD work on Ceph: | |
35 | ================================= | |
36 | ||
37 | pkg install net/ceph-devel | |
38 | ||
39 | Or: | |
40 | cd "place to work on this" | |
41 | git clone https://github.com/wjwithagen/ceph.git | |
42 | cd ceph | |
43 | git checkout wip.FreeBSD | |
44 | ||
45 | Building Ceph | |
46 | ============= | |
47 | - Go and start building | |
48 | ./do_freebsd.sh | |
49 | ||
50 | Parts not (yet) included: | |
51 | ========================= | |
52 | ||
53 | - KRBD | |
54 | Kernel Rados Block Devices is implemented in the Linux kernel | |
55 | And perhaps ggated could be used as a template since it does some of | |
56 | the same, other than just between 2 disks. And it has a userspace | |
57 | counterpart. | |
58 | - BlueStore. | |
59 | FreeBSD and Linux have different AIO API, and that needs to be made | |
60 | compatible Next to that is there discussion in FreeBSD about | |
61 | aio_cancel not working for all devicetypes | |
62 | - CephFS as native filesystem | |
63 | (Ceph-fuse does work.) | |
64 | Cython tries to access an internal field in dirent which does not | |
65 | compile | |
66 | Build Prerequisites | |
67 | =================== | |
68 | ||
69 | Compiling and building Ceph is tested on 12-CURRENT, but I guess/expect | |
70 | 11-RELEASE will also work. And Clang is at 3.8.0. | |
71 | It uses the CLANG toolset that is available, 3.7 is no longer tested, | |
72 | but was working when that was with 11-CURRENT. | |
11fdf7f2 | 73 | Clang 3.4 (on 10.2-STABLE) does not have all required capabilities to |
7c673cae FG |
74 | compile everything |
75 | ||
76 | The following setup will get things running for FreeBSD: | |
77 | ||
78 | This all require root privilidges. | |
79 | ||
80 | - Install bash and link it in /bin | |
81 | sudo pkg install bash | |
82 | sudo ln -s /usr/local/bin/bash /bin/bash | |
83 | ||
84 | Getting the FreeBSD work on Ceph: | |
85 | ================================= | |
86 | ||
87 | - cd "place to work on this" | |
88 | git clone https://github.com/wjwithagen/ceph.git | |
89 | cd ceph | |
90 | git checkout wip.FreeBSD.201702 | |
91 | ||
92 | Building Ceph | |
93 | ============= | |
94 | - Go and start building | |
95 | ./do_freebsd.sh | |
96 | ||
97 | Parts not (yet) included: | |
98 | ========================= | |
99 | ||
100 | - KRBD | |
101 | Kernel Rados Block Devices is implemented in the Linux kernel | |
102 | It seems that there used to be a userspace implementation first. | |
103 | And perhaps ggated could be used as a template since it does some of | |
104 | the same, other than just between 2 disks. And it has a userspace | |
105 | counterpart. | |
106 | - BlueStore. | |
107 | FreeBSD and Linux have different AIO API, and that needs to be made | |
108 | compatible Next to that is there discussion in FreeBSD about | |
109 | aio_cancel not working for all devicetypes | |
110 | - CephFS | |
111 | Cython tries to access an internal field in dirent which does not | |
112 | compile | |
113 | ||
114 | Tests that verify the correct working of the above are also excluded | |
115 | from the testset | |
116 | ||
117 | Tests not (yet) include: | |
118 | ======================= | |
119 | ||
120 | - None, although some test can fail if running tests in parallel and there is | |
121 | not enough swap. Then tests will start to fail in strange ways. | |
122 | ||
123 | Task to do: | |
124 | =========== | |
125 | - Build an automated test platform that will build ceph/master on | |
126 | FreeBSD and report the results back to the Ceph developers. This will | |
127 | increase the maintainability of the FreeBSD side of things. | |
128 | Developers are signalled that they are using Linux-isms that will not | |
129 | compile/run on FreeBSD Ceph has several projects for this: Jenkins, | |
130 | teuthology, pulpito, ... | |
131 | But even just a while { compile } loop and report the build data on a | |
132 | static webpage would do for starters. | |
133 | ||
134 | - Run integration tests to see if the FreeBSD daemons will work with a | |
135 | Linux Ceph platform. | |
136 | ||
137 | - Compile and test the user space RBD (Rados Block Device). | |
138 | ||
139 | - Investigate and see if an in-kernel RBD device could be developed a la | |
140 | 'ggate' | |
141 | ||
142 | - Investigate the keystore, which could be kernel embedded on Linux an | |
143 | currently prevents building Cephfs and some other parts. | |
144 | ||
145 | - Scheduler information is not used atm, because the schedulers work | |
146 | rather different. But at a certain point in time, this would need some | |
147 | attention: | |
148 | in: ./src/common/Thread.cc | |
149 | ||
150 | - Improve the FreeBSD /etc/rc.d initscripts in the Ceph stack. Both | |
151 | for testing, but mainly for running Ceph on production machines. | |
152 | Work on ceph-disk and ceph-deploy to make it more FreeBSD and ZFS | |
153 | compatible. | |
154 | ||
155 | - Build test-cluster and start running some of the teuthology integration | |
156 | tests on these. | |
157 | Teuthology want to build its own libvirt and that does not quite work | |
158 | with all the packages FreeBSD already has in place. Lots of minute | |
159 | details to figure out | |
160 | ||
11fdf7f2 | 161 | - Design a virtual disk implementation that can be used with behyve and |
7c673cae | 162 | attached to an RBD image. |