README.dev

   1 = Setup PVE Development Environment =
   2
   3 0.  Read https://pve.proxmox.com/wiki/Developer_Documentation
   4 1.  Install Debian 9 'stretch' (you can also start from a PVE installation and
   5     skip step 2 - 5, 7 - 11)
   6 2.  Configure the network interface(s)
   7 3.  Change the IP address of your hostname for proper name resolution
   8     in /etc/hosts
   9     Using 127.0.1.1 will not work, so change it to an IP address from your
  10     local network!
  11
  12 4:  Check that the Debian repositories are set properly.
  13     See https://wiki.debian.org/SourcesList for more information.
  14
  15 5.  Optional: Install openssh-server and connect via ssh to the host.
  16
  17     run: apt-get update && apt-get install openssh-server
  18     Connect via ssh to host and switch user to root
  19
  20 6.  Configure 'pvetest' repository in /etc/apt/sources.list.d/:
  21
  22     run: echo "deb http://download.proxmox.com/debian stretch pvetest" > /etc/apt/sources.list.d/pve-development.list
  23
  24 7.  Add the repository key:
  25
  26     run: wget -O- "http://download.proxmox.com/debian/proxmox-ve-release-5.x.gpg" | apt-key add -
  27
  28 8.  run: apt-get update && apt-get dist-upgrade
  29 9.  run: apt-get install proxmox-ve
  30 10. run: mv /etc/apt/sources.list.d/pve-enterprise.list /etc/apt/sources.list.d/pve-enterprise.list.bak
  31
  32 11. You should now have a working Proxmox VE installation.
  33     Open a browser: https://<host_IP_address>:8006 e.g. https://10.0.0.90:8006
  34
  35
  36 = Install build prerequisites for development environment =
  37
  38 NOTE: this is a huge list intended to be able to build (almost) all packages,
  39 from the UI/API components to backend components to our Linux Kernel.
  40 If you only want to hack on specific topics you won't need most of those.
  41 We try to have a complete list of build dependencies in each source
  42 repositories 'debian/control' file. If you run `make deb` dpkg-buildpackage will
  43 stop and tell you if you miss some required packages.
  44
  45 12. For installing the most important, always needed, ones run:
  46
  47 apt-get install build-essential git git-email debhelper pve-doc-generator
  48
  49 Additionally, for quickly installing (almost) all build dependencies run:
  50
  51 apt-get install autotools-dev autogen dh-autoreconf dkms doxygen check pkg-config \
  52 groff quilt dpatch automake autoconf libtool lintian libdevel-cycle-perl \
  53 libjson-perl libcommon-sense-perl liblinux-inotify2-perl libio-stringy-perl \
  54 libstring-shellquote-perl dh-systemd rpm2cpio libsqlite3-dev sqlite3 \
  55 libglib2.0-dev librrd-dev librrds-perl rrdcached libdigest-hmac-perl \
  56 libxml-parser-perl gdb libcrypt-openssl-random-perl \
  57 libcrypt-openssl-rsa-perl libnet-ldap-perl libauthen-pam-perl \
  58 libjson-xs-perl libterm-readline-gnu-perl oathtool libmime-base32-perl \
  59 liboath0 libpci-dev texi2html libsdl1.2-dev libgnutls28-dev \
  60 libspice-protocol-dev xfslibs-dev libnuma-dev libaio-dev \
  61 pve-libspice-server-dev libusbredirparser-dev glusterfs-common \
  62 libusb-1.0-0-dev librbd-dev libpopt-dev iproute bridge-utils numactl \
  63 glusterfs-common ceph-common python-ceph libgoogle-perftools4 \
  64 libfile-chdir-perl lvm2 glusterfs-client liblockfile-simple-perl \
  65 libsystemd-dev libreadline-gplv2-dev libio-multiplex-perl \
  66 libnetfilter-log-dev libipset3 ipset socat libsasl2-dev libogg-dev \
  67 python-pyparsing libfilesys-df-perl libcrypt-ssleay-perl \
  68 libfile-readbackwards-perl libanyevent-perl libanyevent-http-perl \
  69 unzip liblocale-po-perl libfile-sync-perl cstream \
  70 lzop dtach apt-transport-https hdparm gdisk parted ttf-dejavu-core \
  71 liblzma-dev dosfstools mtools libxen-dev libfuse-dev corosync-dev \
  72 libcpg-dev libquorum-dev libcmap-dev libuuid-perl \
  73 libqb-dev libapparmor-dev docbook2x libcap-dev dh-apparmor \
  74 graphviz libseccomp-dev libglib-perl libgtk3-perl libnss3-dev libdlm-dev \
  75 libudev-dev asciidoc-dblatex source-highlight libiscsi-dev libiscsi7 \
  76 librsvg2-bin libarchive-dev libgpgme-dev libcurl4-gnutls-dev \
  77 libtest-mockmodule-perl libjemalloc-dev libjpeg-dev
  78
  79
  80 = Compile PVE packages from Source =
  81
  82 13: Download and install git repositories as Proxmox modules:
  83
  84     run: mkdir /root/proxmox && cd /root/proxmox
  85
  86     run: git clone git://git.proxmox.com/git/pve-common.git
  87
  88     'pve-common.git' is some kind of starting repository and needed for some
  89     other repositories as dependency.
  90     Install this to get an idea of how the installation process is working.
  91
  92     See https://git.proxmox.com/ for all available repositories.
  93
  94 14: Most packages can be installed with 'make dinstall' command.
  95     run: cd pve-common && make dinstall
  96
  97 15: Reboot the system.
  98 16. Learn to use the quilt patch scripts.
  99 17. Happy coding!
 100
 101
 102 = REST vs. SOAP =
 103
 104 We decided to change our SOAP API (1.X) and use a REST like API. The
 105 concept is described in [1] (Resource Oriented Architecture
 106 (ROA)). The main advantage is that we are able to remove a lot of code
 107 (the whole SOAP stack) to reduce software complexity.
 108
 109 We also moved away from server side content generation. Instead we use
 110 the ExtJS Rich Internet Application Framework
 111 (http://www.sencha.com).
 112
 113 That framework, like any other AJAX toolkit, can talk directly to the
 114 REST API using JSON. So we were able to remove the server side
 115 template toolkit completely.
 116
 117 = JSON and JSON Schema =
 118
 119 We use JSON as data format, because it is simple and parse-able by any
 120 web browser.
 121
 122 Additionally, we use JSON Schema [2] to formally describe our API. So
 123 we can automatically generate the whole API Documentation, and we can
 124 verify all parameters and return values.
 125
 126 A great side effect was that we are able to use JSON Schema to
 127 produce command line argument parsers automatically. In fact, the REST
 128 API and the command line tools use the same code.
 129
 130 Object linkage is done using the JSON Hyper Schema (links property).
 131
 132 A small utility called 'pvesh' exposes the whole REST API on the command
 133 line.
 134
 135 So here is a summary of the advantage:
 136
 137    - easy, human readable data format (native web browser format)
 138    - automatic parameter verification (we can also verify return values)
 139    - automatic generation of API documentation
 140    - easy way to create command line tools (using same API).
 141
 142 = API Implementation (PVE::RESTHandler) =
 143
 144 All classes exposing methods on the API use PVE::RESTHandler as base class.
 145
 146   use base qw(PVE::RESTHandler);
 147
 148 To expose methods, one needs to call register_method():
 149
 150   __PACKAGE__->register_method ($schema);
 151
 152 Where $schema is a PVE method schema as described in
 153 PVE::JSONSchema. It includes a description of parameters and return
 154 values, and a reference to the actual code
 155
 156 __PACKAGE__->register_method ({
 157     name => 'echo',
 158     path => 'echo',
 159     method => 'GET',
 160     description => "simple return value of parameter 'text'",
 161     parameters => {
 162         additionalProperties => 0,
 163         properties => {
 164             text => {
 165                  type => 'string',
 166             }
 167         },
 168     },
 169     returns => {
 170         type => 'string',
 171     },
 172     code => sub {
 173         my ($param) = @_;
 174
 175         return $param->{text};
 176     }
 177 });
 178
 179 The 'name' property is only used if you want to call the method
 180 directly from Perl. You can do that using:
 181
 182   print __PACKAGE__->echo({ text => "a test" });
 183
 184 We use Perl's AUTOLOAD feature to implement this. Note: You need to
 185 pass parameters a HASH reference.
 186
 187 There is a special helper method called cli_handler(). This is used by
 188 the CLIHandler Class for command line tools, where you want to pass
 189 arguments as array of strings. This uses Getopt::Long to parse parameters.
 190
 191 There is a second way to map names to methods - using the 'path'
 192 property.  And you can register subclasses. That way you can set up a
 193 filesystem like hierarchy to access methods.
 194
 195 Here is an example:
 196 ----------------------------
 197 package C1;
 198
 199 __PACKAGE__->register_method ({
 200     subclass => "C2",
 201     path => 'sub2',
 202 });
 203
 204
 205 __PACKAGE__->register_method ({
 206     name => 'list1',
 207     path => 'index',
 208     method => 'GET',
 209     ...
 210 });
 211
 212 package C2;
 213
 214 __PACKAGE__->register_method ({
 215     name => 'list2',
 216     path => 'index',
 217     method => 'GET',
 218     ...
 219 });
 220 -------------------------------
 221
 222 The utily method find_handler (in PVE::RESTHandler) can be use to do
 223 'path' related method lookups.
 224
 225 C1->find_handler('GET', "/index")      => C1::list1
 226 C1->find_handler('GET', "/sub2/index") => C2::list2
 227
 228 The HTTP server use the URL (a path) to find the corresponding method.
 229
 230
 231 = References =
 232
 233 [1] RESTful Web Services
 234 Web services for the real world
 235
 236 By
 237     Leonard Richardson, Sam Ruby
 238 Publisher:
 239     O'Reilly Media
 240 Released:
 241     May 2007
 242
 243 [2] JSON Schema links: http://json-schema.org/