]>
Commit | Line | Data |
---|---|---|
95ec8dab MW |
1 | Direct Access for files |
2 | ----------------------- | |
3 | ||
4 | Motivation | |
5 | ---------- | |
6 | ||
7 | The page cache is usually used to buffer reads and writes to files. | |
8 | It is also used to provide the pages which are mapped into userspace | |
9 | by a call to mmap. | |
10 | ||
11 | For block devices that are memory-like, the page cache pages would be | |
12 | unnecessary copies of the original storage. The DAX code removes the | |
13 | extra copy by performing reads and writes directly to the storage device. | |
14 | For file mappings, the storage device is mapped directly into userspace. | |
15 | ||
16 | ||
17 | Usage | |
18 | ----- | |
19 | ||
20 | If you have a block device which supports DAX, you can make a filesystem | |
44f4c054 MW |
21 | on it as usual. The DAX code currently only supports files with a block |
22 | size equal to your kernel's PAGE_SIZE, so you may need to specify a block | |
23 | size when creating the filesystem. When mounting it, use the "-o dax" | |
24 | option on the command line or add 'dax' to the options in /etc/fstab. | |
95ec8dab MW |
25 | |
26 | ||
27 | Implementation Tips for Block Driver Writers | |
28 | -------------------------------------------- | |
29 | ||
30 | To support DAX in your block driver, implement the 'direct_access' | |
31 | block device operation. It is used to translate the sector number | |
32 | (expressed in units of 512-byte sectors) to a page frame number (pfn) | |
33 | that identifies the physical page for the memory. It also returns a | |
34 | kernel virtual address that can be used to access the memory. | |
35 | ||
36 | The direct_access method takes a 'size' parameter that indicates the | |
37 | number of bytes being requested. The function should return the number | |
38 | of bytes that can be contiguously accessed at that offset. It may also | |
39 | return a negative errno if an error occurs. | |
40 | ||
41 | In order to support this method, the storage must be byte-accessible by | |
42 | the CPU at all times. If your device uses paging techniques to expose | |
43 | a large amount of memory through a smaller window, then you cannot | |
44 | implement direct_access. Equally, if your device can occasionally | |
45 | stall the CPU for an extended period, you should also not attempt to | |
46 | implement direct_access. | |
47 | ||
48 | These block devices may be used for inspiration: | |
49 | - axonram: Axon DDR2 device driver | |
50 | - brd: RAM backed block device driver | |
51 | - dcssblk: s390 dcss block device driver | |
221c7dc8 | 52 | - pmem: NVDIMM persistent memory driver |
95ec8dab MW |
53 | |
54 | ||
55 | Implementation Tips for Filesystem Writers | |
56 | ------------------------------------------ | |
57 | ||
58 | Filesystem support consists of | |
59 | - adding support to mark inodes as being DAX by setting the S_DAX flag in | |
60 | i_flags | |
61 | - implementing the direct_IO address space operation, and calling | |
62 | dax_do_io() instead of blockdev_direct_IO() if S_DAX is set | |
63 | - implementing an mmap file operation for DAX files which sets the | |
844f35db MW |
64 | VM_MIXEDMAP and VM_HUGEPAGE flags on the VMA, and setting the vm_ops to |
65 | include handlers for fault, pmd_fault and page_mkwrite (which should | |
66 | probably call dax_fault(), dax_pmd_fault() and dax_mkwrite(), passing the | |
67 | appropriate get_block() callback) | |
95ec8dab | 68 | - calling dax_truncate_page() instead of block_truncate_page() for DAX files |
25726bc1 | 69 | - calling dax_zero_page_range() instead of zero_user() for DAX files |
95ec8dab MW |
70 | - ensuring that there is sufficient locking between reads, writes, |
71 | truncates and page faults | |
72 | ||
73 | The get_block() callback passed to the DAX functions may return | |
74 | uninitialised extents. If it does, it must ensure that simultaneous | |
75 | calls to get_block() (for example by a page-fault racing with a read() | |
76 | or a write()) work correctly. | |
77 | ||
78 | These filesystems may be used for inspiration: | |
221c7dc8 RZ |
79 | - ext2: see Documentation/filesystems/ext2.txt |
80 | - ext4: see Documentation/filesystems/ext4.txt | |
81 | - xfs: see Documentation/filesystems/xfs.txt | |
95ec8dab MW |
82 | |
83 | ||
4b0228fa VV |
84 | Handling Media Errors |
85 | --------------------- | |
86 | ||
87 | The libnvdimm subsystem stores a record of known media error locations for | |
88 | each pmem block device (in gendisk->badblocks). If we fault at such location, | |
89 | or one with a latent error not yet discovered, the application can expect | |
90 | to receive a SIGBUS. Libnvdimm also allows clearing of these errors by simply | |
91 | writing the affected sectors (through the pmem driver, and if the underlying | |
92 | NVDIMM supports the clear_poison DSM defined by ACPI). | |
93 | ||
94 | Since DAX IO normally doesn't go through the driver/bio path, applications or | |
95 | sysadmins have an option to restore the lost data from a prior backup/inbuilt | |
96 | redundancy in the following ways: | |
97 | ||
98 | 1. Delete the affected file, and restore from a backup (sysadmin route): | |
99 | This will free the file system blocks that were being used by the file, | |
100 | and the next time they're allocated, they will be zeroed first, which | |
101 | happens through the driver, and will clear bad sectors. | |
102 | ||
103 | 2. Truncate or hole-punch the part of the file that has a bad-block (at least | |
104 | an entire aligned sector has to be hole-punched, but not necessarily an | |
105 | entire filesystem block). | |
106 | ||
107 | These are the two basic paths that allow DAX filesystems to continue operating | |
108 | in the presence of media errors. More robust error recovery mechanisms can be | |
109 | built on top of this in the future, for example, involving redundancy/mirroring | |
110 | provided at the block layer through DM, or additionally, at the filesystem | |
111 | level. These would have to rely on the above two tenets, that error clearing | |
112 | can happen either by sending an IO through the driver, or zeroing (also through | |
113 | the driver). | |
114 | ||
115 | ||
95ec8dab MW |
116 | Shortcomings |
117 | ------------ | |
118 | ||
119 | Even if the kernel or its modules are stored on a filesystem that supports | |
120 | DAX on a block device that supports DAX, they will still be copied into RAM. | |
121 | ||
d92576f1 MW |
122 | The DAX code does not work correctly on architectures which have virtually |
123 | mapped caches such as ARM, MIPS and SPARC. | |
124 | ||
95ec8dab MW |
125 | Calling get_user_pages() on a range of user memory that has been mmaped |
126 | from a DAX file will fail as there are no 'struct page' to describe | |
127 | those pages. This problem is being worked on. That means that O_DIRECT | |
128 | reads/writes to those memory ranges from a non-DAX file will fail (note | |
129 | that O_DIRECT reads/writes _of a DAX file_ do work, it is the memory | |
130 | that is being accessed that is key here). Other things that will not | |
131 | work include RDMA, sendfile() and splice(). |