]>
Commit | Line | Data |
---|---|---|
f67539c2 TL |
1 | # Apache Thrift - integration test suite |
2 | ||
3 | This is the cross everything integration test suite for Apache Thrift. | |
4 | ||
5 | ## Run | |
6 | ||
7 | ### A. Using Make | |
8 | ||
9 | The test can be executed by: | |
10 | ||
11 | make cross | |
12 | ||
13 | This starts the [test.py](test.py) script which does the real cross test with | |
14 | different transports, protocols and languages. | |
15 | ||
16 | Note that this skips any language that is not built locally. It also skips | |
17 | tests that are known to be failing. If you need more control over which tests | |
18 | to run, read following section. | |
19 | ||
20 | ### B. Using test script directly | |
21 | ||
22 | Alternatively, you can invoke [test.py](test.py) directly. You need to run`make | |
23 | precross` once before executing it for the first time. | |
24 | ||
25 | For example, if you changed something in `nodejs` library and need to verify | |
26 | the patch, you can skip everything except `nodejs` itself and some reference | |
27 | implementation (currently `cpp` and `java` are recommended) like this: | |
28 | ||
29 | ./configure --without-c_glib -without-csharp --without-erlang --without-lua ... | |
30 | make precross -j8 | |
31 | test/test.py --server cpp,java --client nodejs | |
32 | test/test.py --server nodejs --client cpp,java | |
33 | ||
34 | Another useful flag is --regex. For example, to run all tests that involve | |
35 | Java TBinaryProtocol: | |
36 | ||
37 | test/test.py --regex "java.*binary" | |
38 | ||
39 | ## Test case definition file | |
40 | ||
41 | The cross test cases are defined in [tests.json](tests.json). | |
42 | The root element is collection of test target definitions. | |
43 | Each test target definition looks like this: | |
44 | ||
45 | { | |
46 | "name": "somelib", | |
47 | ||
48 | "client": { | |
49 | "command": ["somelib_client_executable"], | |
50 | "workdir": "somelib/bin", | |
51 | "protocols": ["binary"], | |
52 | "transports": ["buffered"], | |
53 | "sockets": ["ip"], | |
54 | }, | |
55 | "server": { | |
56 | "command": ["somelib_server_executable"], | |
57 | "workdir": "somelib/bin", | |
58 | "protocols": ["binary"], | |
59 | "transports": ["buffered"], | |
60 | "sockets": ["ip", "ip-ssl"], | |
61 | } | |
62 | } | |
63 | ||
64 | Either client or server definition or both should be present. | |
65 | ||
66 | Parameters that are common to both `client` and `server` can be put to target | |
67 | definition root: | |
68 | ||
69 | { | |
70 | "name": "somelib", | |
71 | ||
72 | "workdir": "somelib/bin", | |
73 | "protocols": ["binary"], | |
74 | "transports": ["buffered"], | |
75 | "sockets": ["ip"], | |
76 | ||
77 | "client": { "command": ["somelib_client_executable"] }, | |
78 | "server": { | |
79 | "command": ["somelib_server_executable"], | |
80 | "sockets": ["ip-ssl"] | |
81 | } | |
82 | } | |
83 | ||
84 | For the complete list of supported keys and their effect, see source code | |
85 | comment at the opt of [crossrunner/collect.py](crossrunner/collect.py). | |
86 | ||
87 | ||
88 | ## List of known failures | |
89 | ||
90 | Since many cross tests currently fail (mainly due to partial incompatibility | |
91 | around exception handling), the test script specifically report for "not known | |
92 | before" failures. | |
93 | ||
94 | For this purpose, test cases known to (occasionally) fail are listed in | |
95 | `known_failures_<platform>.json` where `<platform>` matches with python | |
96 | `platform.system()` string. | |
97 | ||
98 | Currently, only Linux version is included. | |
99 | ||
100 | FYI, the file is initially generated by | |
101 | ||
102 | test/test.py --update-expected-failures=overwrite | |
103 | ||
104 | after a full test run, then repeatedly | |
105 | ||
106 | test/test.py --skip-known-failures | |
107 | test/test.py --update-expected-failures=merge | |
108 | ||
109 | to update the known failures, run | |
110 | ||
111 | make fail | |
112 | ||
113 | ## Test executable specification | |
114 | ||
115 | ### Command line parameters | |
116 | ||
117 | Unit tests for languages are usually located under lib/<lang>/test/ | |
118 | cross language tests according to [ThriftTest.thrift](ThriftTest.thrift) shall be | |
119 | provided for every language including executables with the following command | |
120 | line interface: | |
121 | ||
122 | **Server command line interface:** | |
123 | ||
124 | $ ./cpp/TestServer -h | |
125 | Allowed options: | |
126 | -h [ --help ] produce help message | |
127 | --port arg (=9090) Port number to listen | |
128 | --domain-socket arg Unix Domain Socket (e.g. /tmp/ThriftTest.thrift) | |
129 | --named-pipe arg Windows Named Pipe (e.g. MyThriftPipe) | |
130 | --server-type arg (=simple) type of server, "simple", "thread-pool", | |
131 | "threaded", or "nonblocking" | |
132 | --transport arg (=buffered) transport: buffered, framed, http, anonpipe | |
133 | --protocol arg (=binary) protocol: binary, compact, json | |
134 | --ssl Encrypted Transport using SSL | |
135 | --processor-events processor-events | |
136 | -n [ --workers ] arg (=4) Number of thread pools workers. Only valid for | |
137 | thread-pool server type | |
138 | ||
139 | **Client command line interface:** | |
140 | ||
141 | $ ./cpp/TestClient -h | |
142 | Allowed options: | |
143 | -h [ --help ] produce help message | |
144 | --host arg (=localhost) Host to connect | |
145 | --port arg (=9090) Port number to connect | |
146 | --domain-socket arg Domain Socket (e.g. /tmp/ThriftTest.thrift), | |
147 | instead of host and port | |
148 | --named-pipe arg Windows Named Pipe (e.g. MyThriftPipe) | |
149 | --anon-pipes hRead hWrite Windows Anonymous Pipes pair (handles) | |
150 | --transport arg (=buffered) Transport: buffered, framed, http, evhttp | |
151 | --protocol arg (=binary) Protocol: binary, compact, json | |
152 | --ssl Encrypted Transport using SSL | |
153 | -n [ --testloops ] arg (=1) Number of Tests | |
154 | -t [ --threads ] arg (=1) Number of Test threads | |
155 | ||
156 | If you have executed the **make check** or **make cross** then you will be able to browse | |
157 | [gen-html/ThriftTest.html](gen-html/ThriftTest.html) with the test documentation. | |
158 | ||
159 | ### Return code | |
160 | ||
161 | The return code (exit code) shall be 0 on success, or an integer in the range 1 - 255 on errors. | |
162 | In order to signal failed tests, the return code shall be composed from these bits to indicate | |
163 | failing tests: | |
164 | ||
165 | #define TEST_BASETYPES 1 // 0000 0001 | |
166 | #define TEST_STRUCTS 2 // 0000 0010 | |
167 | #define TEST_CONTAINERS 4 // 0000 0100 | |
168 | #define TEST_EXCEPTIONS 8 // 0000 1000 | |
169 | #define TEST_UNKNOWN 64 // 0100 0000 (Failed to prepare environment etc.) | |
170 | #define TEST_TIMEOUT 128 // 1000 0000 | |
171 | #define TEST_NOTUSED 48 // 0011 0000 (reserved bits) | |
172 | ||
173 | Tests that have not been executed at all count as errors. | |
174 | ||
175 | **Example:** | |
176 | ||
177 | During tests, the test client notices that some of the Struct tests fail. | |
178 | Furthermore, due to some other problem none of the Exception tests is executed. | |
179 | Therefore, the test client returns the code `10 = 2 | 8`, indicating the failure | |
180 | of both test 2 (TEST_STRUCTS) and test 8 (TEST_EXCEPTIONS). | |
181 | ||
182 | ||
183 | ## SSL | |
184 | Test Keys and Certificates are provided in multiple formats under the following | |
185 | directory [test/keys](keys) |