ceph/doc/dev/corpus.rst

   1
   2 Corpus structure
   3 ================
   4
   5 ceph.git/ceph-object-corpus is a submodule.::
   6
   7  bin/   # misc scripts
   8  archive/$version/objects/$type/$hash  # a sample of encoded objects from a specific version
   9
  10 You can also mark known or deliberate incompatibilities between versions with::
  11
  12  archive/$version/forward_incompat/$type
  13
  14 The presence of a file indicates that new versions of code cannot
  15 decode old objects across that ``$version`` (this is normally the case).
  16
  17
  18 How to generate an object corpus
  19 --------------------------------
  20
  21 .. highlight:: shell
  22
  23 We can generate an object corpus for a particular version of ceph using the
  24 script of ``script/gen-corpus.sh``, or by following the instructions below:
  25
  26 #. Checkout a clean repo (best not to do this where you normally work)::
  27
  28         git clone ceph.git
  29         cd ceph
  30         git submodule update --init --recursive
  31
  32 #. Build with flag to dump objects to ``/tmp/foo``::
  33
  34         rm -rf /tmp/foo ; mkdir /tmp/foo
  35         do_cmake.sh -DCMAKE_CXX_FLAGS="-DENCODE_DUMP_PATH=/tmp/foo"
  36         cd build
  37         make
  38
  39 #. Start via vstart::
  40
  41         cd build
  42         MON=3 MGR=2 OSD=3 MDS=3 RGW=1 ../src/vstart.sh -n -x
  43
  44 #. Use as much functionality of the cluster as you can, to exercise as many object encoder methods as possible::
  45
  46         bin/ceph osd pool create mypool
  47         bin/rados -p mypool bench 10 write -b 123
  48         bin/ceph osd out 0
  49         bin/ceph osd in 0
  50         bin/init-ceph restart osd.1
  51         for f in ../qa/workunits/cls/*.sh ; do PATH="bin:$PATH" $f ; done
  52         PATH="bin:$PATH" ../qa/workunits/rados/test.sh
  53         bin/ceph_test_librbd
  54         bin/ceph_test_libcephfs
  55         bin/init-ceph restart mds.a
  56         ../qa/workunits/rgw/run-s3tests.sh
  57
  58 #. Stop::
  59
  60         ../src/stop.sh
  61
  62 #. Import the corpus (this will take a few minutes)::
  63
  64         ../src/test/encoding/import.sh /tmp/foo `bin/ceph-dencoder version` ../ceph-object-corpus/archive
  65         ../src/test/encoding/import-generated.sh ../ceph-object-corpus/archive
  66
  67 #. Prune it!  There will be a bazillion copies of various objects, and we only want a representative sample.::
  68
  69         pushd ../ceph-object-corpus
  70         bin/prune-archive.sh
  71         popd
  72
  73 #. Verify the tests pass::
  74
  75         ctest -R readable.sh
  76
  77 #. Commit it to the corpus repo and push::
  78
  79         pushd ../ceph-object-corpus
  80         git checkout -b wip-new
  81         git add archive/`../build/bin/ceph-dencoder version`
  82         git commit -m `../build/bin/ceph-dencoder version`
  83         git remote add cc git@github.com:ceph/ceph-object-corpus.git
  84         git push cc wip-new
  85         popd
  86
  87 #. Go test it out::
  88
  89         cd my/regular/tree
  90         cd ceph-object-corpus
  91         git fetch origin
  92         git checkout wip-new
  93         cd ../build
  94         ctest -R readable.sh
  95
  96 #. If everything looks good, update the submodule master branch, and commit the submodule in ceph.git.
  97
  98
  99
 100