README.dev

   1 = Setup PVE Development Environment =
   2
   3 0.  Read https://pve.proxmox.com/wiki/Developer_Documentation
   4 1.  Install Debian 9 'stretch' (you can also start from a PVE installation and
   5     skip step 2 - 5, 7 - 11)
   6 2.  Configure the network interface(s)
   7 3.  Change the IP address of your hostname for proper name resolution
   8     in /etc/hosts
   9     Using 127.0.1.1 will not work, so change it to an IP address from your
  10     local network!
  11
  12 4:  Check that the Debian repositories are set properly.
  13     See https://wiki.debian.org/SourcesList for more information.
  14
  15 5.  Optional: Install openssh-server and connect via ssh to the host.
  16
  17     run: apt-get update && apt-get install openssh-server
  18     Connect via ssh to host and switch user to root
  19
  20 6.  Configure 'pvetest' repository in /etc/apt/sources.list.d/:
  21
  22     run: echo "deb http://download.proxmox.com/debian stretch pvetest" > /etc/apt/sources.list.d/pve-development.list
  23
  24 7.  Add the repository key:
  25
  26     run: wget -O- "http://download.proxmox.com/debian/proxmox-ve-release-5.x.gpg" | apt-key add -
  27
  28 8.  run: apt-get update && apt-get dist-upgrade
  29 9.  run: apt-get install proxmox-ve
  30 10. run: mv /etc/apt/sources.list.d/pve-enterprise.list /etc/apt/sources.list.d/pve-enterprise.list.bak
  31
  32 11. You should now have a working Proxmox VE installation.
  33     Open a browser: https://<host_IP_address>:8006 e.g. https://10.0.0.90:8006
  34
  35
  36 = Install build prerequisites for development environment =
  37
  38 NOTE: this is a huge and probably outdated list intended to be able to build
  39 (almost) all packages, from the UI/API components to backend components to our
  40 Linux Kernel.  If you only want to hack on specific topics you won't need most
  41 of those.
  42 Instead we try to have a complete list of build dependencies in each source
  43 repositories 'debian/control' file. If you run `make deb` dpkg-buildpackage
  44 will stop and tell you if you miss some required packages.
  45
  46 12. For installing the most important, always needed, ones run:
  47
  48 apt-get install build-essential git git-email debhelper pve-doc-generator
  49
  50 Additionally, for quickly installing (almost) all build dependencies run:
  51
  52 WARNING: this list is almost for sure outdated! Use the build-deps definitions
  53 defined in each package! You could install `devscripts` (huge package, but nice
  54 helpers) and use:
  55 # mk-build-deps --install
  56 in the top-level directory of a git repository.
  57
  58 apt-get install autotools-dev autogen dh-autoreconf dkms doxygen check pkg-config \
  59 groff quilt dpatch automake autoconf libtool lintian libdevel-cycle-perl \
  60 libjson-perl libcommon-sense-perl liblinux-inotify2-perl libio-stringy-perl \
  61 libstring-shellquote-perl dh-systemd rpm2cpio libsqlite3-dev sqlite3 \
  62 libglib2.0-dev librrd-dev librrds-perl rrdcached libdigest-hmac-perl \
  63 libxml-parser-perl gdb libcrypt-openssl-random-perl \
  64 libcrypt-openssl-rsa-perl libnet-ldap-perl libauthen-pam-perl \
  65 libjson-xs-perl libterm-readline-gnu-perl oathtool libmime-base32-perl \
  66 liboath0 libpci-dev texi2html libsdl1.2-dev libgnutls28-dev \
  67 libspice-protocol-dev xfslibs-dev libnuma-dev libaio-dev \
  68 pve-libspice-server-dev libusbredirparser-dev glusterfs-common \
  69 libusb-1.0-0-dev librbd-dev libpopt-dev iproute bridge-utils numactl \
  70 glusterfs-common ceph-common python-ceph libgoogle-perftools4 \
  71 libfile-chdir-perl lvm2 glusterfs-client liblockfile-simple-perl \
  72 libsystemd-dev libreadline-gplv2-dev libio-multiplex-perl \
  73 libnetfilter-log-dev libipset3 ipset socat libsasl2-dev libogg-dev \
  74 python-pyparsing libfilesys-df-perl libcrypt-ssleay-perl \
  75 libfile-readbackwards-perl libanyevent-perl libanyevent-http-perl \
  76 unzip liblocale-po-perl libfile-sync-perl cstream \
  77 lzop dtach hdparm gdisk parted ttf-dejavu-core \
  78 liblzma-dev dosfstools mtools libxen-dev libfuse-dev libcpg-dev libquorum-dev \
  79 libcmap-dev libuuid-perl libqb-dev libapparmor-dev docbook2x libcap-dev \
  80 dh-apparmor graphviz libseccomp-dev libglib-perl libgtk3-perl libnss3-dev \
  81 libdlm-dev libudev-dev asciidoc-dblatex source-highlight libiscsi-dev \
  82 libiscsi7 librsvg2-bin libarchive-dev libgpgme-dev libcurl4-gnutls-dev \
  83 libtest-mockmodule-perl libjemalloc-dev libjpeg-dev
  84
  85
  86 = Compile PVE packages from Source =
  87
  88 13: Download and install git repositories as Proxmox modules:
  89
  90     run: mkdir /root/proxmox && cd /root/proxmox
  91
  92     run: git clone git://git.proxmox.com/git/pve-common.git
  93
  94     'pve-common.git' is some kind of starting repository and needed for some
  95     other repositories as dependency.
  96     Install this to get an idea of how the installation process is working.
  97
  98     See https://git.proxmox.com/ for all available repositories.
  99
 100 14: Most packages can be installed with 'make dinstall' command.
 101     run: cd pve-common && make dinstall
 102
 103 15: Reboot the system.
 104 16. Learn to use the quilt patch scripts.
 105 17. Happy coding!
 106
 107
 108 = REST vs. SOAP =
 109
 110 We decided to change our SOAP API (1.X) and use a REST like API. The
 111 concept is described in [1] (Resource Oriented Architecture
 112 (ROA)). The main advantage is that we are able to remove a lot of code
 113 (the whole SOAP stack) to reduce software complexity.
 114
 115 We also moved away from server side content generation. Instead we use
 116 the ExtJS Rich Internet Application Framework
 117 (http://www.sencha.com).
 118
 119 That framework, like any other AJAX toolkit, can talk directly to the
 120 REST API using JSON. So we were able to remove the server side
 121 template toolkit completely.
 122
 123 = JSON and JSON Schema =
 124
 125 We use JSON as data format, because it is simple and parse-able by any
 126 web browser.
 127
 128 Additionally, we use JSON Schema [2] to formally describe our API. So
 129 we can automatically generate the whole API Documentation, and we can
 130 verify all parameters and return values.
 131
 132 A great side effect was that we are able to use JSON Schema to
 133 produce command line argument parsers automatically. In fact, the REST
 134 API and the command line tools use the same code.
 135
 136 Object linkage is done using the JSON Hyper Schema (links property).
 137
 138 A small utility called 'pvesh' exposes the whole REST API on the command
 139 line.
 140
 141 So here is a summary of the advantage:
 142
 143    - easy, human readable data format (native web browser format)
 144    - automatic parameter verification (we can also verify return values)
 145    - automatic generation of API documentation
 146    - easy way to create command line tools (using same API).
 147
 148 = API Implementation (PVE::RESTHandler) =
 149
 150 All classes exposing methods on the API use PVE::RESTHandler as base class.
 151
 152   use base qw(PVE::RESTHandler);
 153
 154 To expose methods, one needs to call register_method():
 155
 156   __PACKAGE__->register_method ($schema);
 157
 158 Where $schema is a PVE method schema as described in
 159 PVE::JSONSchema. It includes a description of parameters and return
 160 values, and a reference to the actual code
 161
 162 __PACKAGE__->register_method ({
 163     name => 'echo',
 164     path => 'echo',
 165     method => 'GET',
 166     description => "simple return value of parameter 'text'",
 167     parameters => {
 168         additionalProperties => 0,
 169         properties => {
 170             text => {
 171                  type => 'string',
 172             }
 173         },
 174     },
 175     returns => {
 176         type => 'string',
 177     },
 178     code => sub {
 179         my ($param) = @_;
 180
 181         return $param->{text};
 182     }
 183 });
 184
 185 The 'name' property is only used if you want to call the method
 186 directly from Perl. You can do that using:
 187
 188   print __PACKAGE__->echo({ text => "a test" });
 189
 190 We use Perl's AUTOLOAD feature to implement this. Note: You need to
 191 pass parameters a HASH reference.
 192
 193 There is a special helper method called cli_handler(). This is used by
 194 the CLIHandler Class for command line tools, where you want to pass
 195 arguments as array of strings. This uses Getopt::Long to parse parameters.
 196
 197 There is a second way to map names to methods - using the 'path'
 198 property.  And you can register subclasses. That way you can set up a
 199 filesystem like hierarchy to access methods.
 200
 201 Here is an example:
 202 ----------------------------
 203 package C1;
 204
 205 __PACKAGE__->register_method ({
 206     subclass => "C2",
 207     path => 'sub2',
 208 });
 209
 210
 211 __PACKAGE__->register_method ({
 212     name => 'list1',
 213     path => 'index',
 214     method => 'GET',
 215     ...
 216 });
 217
 218 package C2;
 219
 220 __PACKAGE__->register_method ({
 221     name => 'list2',
 222     path => 'index',
 223     method => 'GET',
 224     ...
 225 });
 226 -------------------------------
 227
 228 The utily method find_handler (in PVE::RESTHandler) can be use to do
 229 'path' related method lookups.
 230
 231 C1->find_handler('GET', "/index")      => C1::list1
 232 C1->find_handler('GET', "/sub2/index") => C2::list2
 233
 234 The HTTP server use the URL (a path) to find the corresponding method.
 235
 236
 237 = References =
 238
 239 [1] RESTful Web Services
 240 Web services for the real world
 241
 242 By
 243     Leonard Richardson, Sam Ruby
 244 Publisher:
 245     O'Reilly Media
 246 Released:
 247     May 2007
 248
 249 [2] JSON Schema links: http://json-schema.org/