]>
Commit | Line | Data |
---|---|---|
d78345bc DM |
1 | //! This module implements the proxmox backup chunked data storage |
2 | //! | |
3 | //! A chunk is simply defined as binary blob. We store them inside a | |
4 | //! `ChunkStore`, addressed by the SHA256 digest of the binary | |
5 | //! blob. This technology is also known as content-addressable | |
6 | //! storage. | |
7 | //! | |
8 | //! We store larger files by splitting them into chunks. The resulting | |
9 | //! SHA256 digest list is stored as separate index file. The | |
10 | //! `DynamicIndex*` format is able to deal with dynamic chunk sizes, | |
11 | //! whereas the `FixedIndex*` format is an optimization to store a | |
12 | //! list of equal sized chunks. | |
04652189 DM |
13 | //! |
14 | //! # ChunkStore Locking | |
15 | //! | |
16 | //! We need to be able to restart the proxmox-backup service daemons, | |
17 | //! so that we can update the software without rebooting the host. But | |
18 | //! such restarts must not abort running backup jobs, so we need to | |
19 | //! keep the old service running until those jobs are finished. This | |
20 | //! implies that we need some kink of locking for the | |
21 | //! ChunkStore. Please note that it is perfectly valid to have | |
22 | //! multiple parallel ChunkStore writers, even when they write the | |
23 | //! same chunk (because the chunk would have the same name and the | |
24 | //! same data). The only real problem is garbage collection, because | |
25 | //! we need to avoid deleting chunks which are still referenced. | |
26 | //! | |
27 | //! * Read Index Files: | |
28 | //! | |
29 | //! Acquire shared lock for .idx files. | |
30 | //! | |
31 | //! | |
32 | //! * Delete Index Files: | |
33 | //! | |
34 | //! Acquire exclusive lock for .idx files. This makes sure that we do | |
35 | //! not delete index files while they are still in use. | |
36 | //! | |
37 | //! | |
38 | //! * Create Index Files: | |
39 | //! | |
40 | //! Acquire shared lock for ChunkStore. | |
41 | //! | |
42 | //! Note: We create temporary (.tmp) file, then do an atomic rename ... | |
43 | //! | |
44 | //! | |
45 | //! * Garbage Collect: | |
46 | //! | |
47 | //! Acquire exclusive lock for ChunkStore. | |
48 | //! | |
49 | //! | |
50 | //! * Server Restart | |
51 | //! | |
52 | //! Try to abort running garbage collection to release exclusive | |
53 | //! ChunkStore lock asap. Start new service with existing listening | |
54 | //! socket. | |
55 | //! | |
cbdd8c54 | 56 | |
7e336555 DM |
57 | mod chunk_stat; |
58 | pub use chunk_stat::*; | |
59 | ||
06178f13 | 60 | pub use proxmox_protocol::Chunker; |
e5064ba6 DM |
61 | |
62 | mod chunk_store; | |
63 | pub use chunk_store::*; | |
64 | ||
7bc1d727 WB |
65 | mod index; |
66 | pub use index::*; | |
67 | ||
e5064ba6 DM |
68 | mod fixed_index; |
69 | pub use fixed_index::*; | |
70 | ||
71 | mod dynamic_index; | |
72 | pub use dynamic_index::*; | |
73 | ||
b3483782 DM |
74 | mod backup_info; |
75 | pub use backup_info::*; | |
76 | ||
e5064ba6 DM |
77 | mod datastore; |
78 | pub use datastore::*; |