]>
Commit | Line | Data |
---|---|---|
114a0a08 BS |
1 | |
2 | Fault Injection | |
3 | =============== | |
4 | Fault injection is a method for forcing errors that may not normally occur, or | |
5 | may be difficult to reproduce. Forcing these errors in a controlled environment | |
6 | can help the developer find and fix bugs before their code is shipped in a | |
7 | production system. Injecting an error on the Linux NFS server will allow us to | |
8 | observe how the client reacts and if it manages to recover its state correctly. | |
9 | ||
10 | NFSD_FAULT_INJECTION must be selected when configuring the kernel to use this | |
11 | feature. | |
12 | ||
13 | ||
14 | Using Fault Injection | |
15 | ===================== | |
16 | On the client, mount the fault injection server through NFS v4.0+ and do some | |
17 | work over NFS (open files, take locks, ...). | |
18 | ||
19 | On the server, mount the debugfs filesystem to <debug_dir> and ls | |
20 | <debug_dir>/nfsd. This will show a list of files that will be used for | |
21 | injecting faults on the NFS server. As root, write a number n to the file | |
22 | corresponding to the action you want the server to take. The server will then | |
23 | process the first n items it finds. So if you want to forget 5 locks, echo '5' | |
24 | to <debug_dir>/nfsd/forget_locks. A value of 0 will tell the server to forget | |
25 | all corresponding items. A log message will be created containing the number | |
26 | of items forgotten (check dmesg). | |
27 | ||
28 | Go back to work on the client and check if the client recovered from the error | |
29 | correctly. | |
30 | ||
31 | ||
32 | Available Faults | |
33 | ================ | |
34 | forget_clients: | |
35 | The NFS server keeps a list of clients that have placed a mount call. If | |
36 | this list is cleared, the server will have no knowledge of who the client | |
37 | is, forcing the client to reauthenticate with the server. | |
38 | ||
39 | forget_openowners: | |
40 | The NFS server keeps a list of what files are currently opened and who | |
41 | they were opened by. Clearing this list will force the client to reopen | |
42 | its files. | |
43 | ||
44 | forget_locks: | |
45 | The NFS server keeps a list of what files are currently locked in the VFS. | |
46 | Clearing this list will force the client to reclaim its locks (files are | |
47 | unlocked through the VFS as they are cleared from this list). | |
48 | ||
49 | forget_delegations: | |
50 | A delegation is used to assure the client that a file, or part of a file, | |
51 | has not changed since the delegation was awarded. Clearing this list will | |
52 | force the client to reaquire its delegation before accessing the file | |
53 | again. | |
54 | ||
55 | recall_delegations: | |
56 | Delegations can be recalled by the server when another client attempts to | |
57 | access a file. This test will notify the client that its delegation has | |
58 | been revoked, forcing the client to reaquire the delegation before using | |
59 | the file again. | |
60 | ||
61 | ||
62 | tools/nfs/inject_faults.sh script | |
63 | ================================= | |
64 | This script has been created to ease the fault injection process. This script | |
65 | will detect the mounted debugfs directory and write to the files located there | |
66 | based on the arguments passed by the user. For example, running | |
67 | `inject_faults.sh forget_locks 1` as root will instruct the server to forget | |
68 | one lock. Running `inject_faults forget_locks` will instruct the server to | |
69 | forgetall locks. |