]>
Commit | Line | Data |
---|---|---|
0a29b90c FG |
1 | # [base64](https://crates.io/crates/base64) |
2 | ||
3 | [![](https://img.shields.io/crates/v/base64.svg)](https://crates.io/crates/base64) [![Docs](https://docs.rs/base64/badge.svg)](https://docs.rs/base64) [![CircleCI](https://circleci.com/gh/marshallpierce/rust-base64/tree/master.svg?style=shield)](https://circleci.com/gh/marshallpierce/rust-base64/tree/master) [![codecov](https://codecov.io/gh/marshallpierce/rust-base64/branch/master/graph/badge.svg)](https://codecov.io/gh/marshallpierce/rust-base64) [![unsafe forbidden](https://img.shields.io/badge/unsafe-forbidden-success.svg)](https://github.com/rust-secure-code/safety-dance/) | |
4 | ||
5 | <a href="https://www.jetbrains.com/?from=rust-base64"><img src="/icon_CLion.svg" height="40px"/></a> | |
6 | ||
7 | Made with CLion. Thanks to JetBrains for supporting open source! | |
8 | ||
9 | It's base64. What more could anyone want? | |
10 | ||
11 | This library's goals are to be *correct* and *fast*. It's thoroughly tested and widely used. It exposes functionality at | |
12 | multiple levels of abstraction so you can choose the level of convenience vs performance that you want, | |
13 | e.g. `decode_engine_slice` decodes into an existing `&mut [u8]` and is pretty fast (2.6GiB/s for a 3 KiB input), | |
14 | whereas `decode_engine` allocates a new `Vec<u8>` and returns it, which might be more convenient in some cases, but is | |
15 | slower (although still fast enough for almost any purpose) at 2.1 GiB/s. | |
16 | ||
17 | See the [docs](https://docs.rs/base64) for all the details. | |
18 | ||
19 | ## FAQ | |
20 | ||
21 | ### I need to decode base64 with whitespace/null bytes/other random things interspersed in it. What should I do? | |
22 | ||
23 | Remove non-base64 characters from your input before decoding. | |
24 | ||
25 | If you have a `Vec` of base64, [retain](https://doc.rust-lang.org/std/vec/struct.Vec.html#method.retain) can be used to | |
26 | strip out whatever you need removed. | |
27 | ||
28 | If you have a `Read` (e.g. reading a file or network socket), there are various approaches. | |
29 | ||
30 | - Use [iter_read](https://crates.io/crates/iter-read) together with `Read`'s `bytes()` to filter out unwanted bytes. | |
31 | - Implement `Read` with a `read()` impl that delegates to your actual `Read`, and then drops any bytes you don't want. | |
32 | ||
33 | ### I need to line-wrap base64, e.g. for MIME/PEM. | |
34 | ||
35 | [line-wrap](https://crates.io/crates/line-wrap) does just that. | |
36 | ||
37 | ### I want canonical base64 encoding/decoding. | |
38 | ||
39 | First, don't do this. You should no more expect Base64 to be canonical than you should expect compression algorithms to | |
40 | produce canonical output across all usage in the wild (hint: they don't). | |
41 | However, [people are drawn to their own destruction like moths to a flame](https://eprint.iacr.org/2022/361), so here we | |
42 | are. | |
43 | ||
44 | There are two opportunities for non-canonical encoding (and thus, detection of the same during decoding): the final bits | |
45 | of the last encoded token in two or three token suffixes, and the `=` token used to inflate the suffix to a full four | |
46 | tokens. | |
47 | ||
48 | The trailing bits issue is unavoidable: with 6 bits available in each encoded token, 1 input byte takes 2 tokens, | |
49 | with the second one having some bits unused. Same for two input bytes: 16 bits, but 3 tokens have 18 bits. Unless we | |
50 | decide to stop shipping whole bytes around, we're stuck with those extra bits that a sneaky or buggy encoder might set | |
51 | to 1 instead of 0. | |
52 | ||
53 | The `=` pad bytes, on the other hand, are entirely a self-own by the Base64 standard. They do not affect decoding other | |
54 | than to provide an opportunity to say "that padding is incorrect". Exabytes of storage and transfer have no doubt been | |
55 | wasted on pointless `=` bytes. Somehow we all seem to be quite comfortable with, say, hex-encoded data just stopping | |
56 | when it's done rather than requiring a confirmation that the author of the encoder could count to four. Anyway, there | |
57 | are two ways to make pad bytes predictable: require canonical padding to the next multiple of four bytes as per the RFC, | |
58 | or, if you control all producers and consumers, save a few bytes by requiring no padding (especially applicable to the | |
59 | url-safe alphabet). | |
60 | ||
61 | All `Engine` implementations must at a minimum support treating non-canonical padding of both types as an error, and | |
62 | optionally may allow other behaviors. | |
63 | ||
64 | ## Rust version compatibility | |
65 | ||
781aab86 | 66 | The minimum supported Rust version is 1.48.0. |
0a29b90c FG |
67 | |
68 | # Contributing | |
69 | ||
70 | Contributions are very welcome. However, because this library is used widely, and in security-sensitive contexts, all | |
71 | PRs will be carefully scrutinized. Beyond that, this sort of low level library simply needs to be 100% correct. Nobody | |
72 | wants to chase bugs in encoding of any sort. | |
73 | ||
74 | All this means that it takes me a fair amount of time to review each PR, so it might take quite a while to carve out the | |
75 | free time to give each PR the attention it deserves. I will get to everyone eventually! | |
76 | ||
77 | ## Developing | |
78 | ||
fe692bf9 | 79 | Benchmarks are in `benches/`. |
0a29b90c FG |
80 | |
81 | ```bash | |
fe692bf9 | 82 | cargo bench |
0a29b90c FG |
83 | ``` |
84 | ||
85 | ## no_std | |
86 | ||
87 | This crate supports no_std. By default the crate targets std via the `std` feature. You can deactivate | |
88 | the `default-features` to target `core` instead. In that case you lose out on all the functionality revolving | |
89 | around `std::io`, `std::error::Error`, and heap allocations. There is an additional `alloc` feature that you can activate | |
90 | to bring back the support for heap allocations. | |
91 | ||
92 | ## Profiling | |
93 | ||
94 | On Linux, you can use [perf](https://perf.wiki.kernel.org/index.php/Main_Page) for profiling. Then compile the | |
fe692bf9 | 95 | benchmarks with `cargo bench --no-run`. |
0a29b90c FG |
96 | |
97 | Run the benchmark binary with `perf` (shown here filtering to one particular benchmark, which will make the results | |
98 | easier to read). `perf` is only available to the root user on most systems as it fiddles with event counters in your | |
99 | CPU, so use `sudo`. We need to run the actual benchmark binary, hence the path into `target`. You can see the actual | |
fe692bf9 | 100 | full path with `cargo bench -v`; it will print out the commands it runs. If you use the exact path |
0a29b90c FG |
101 | that `bench` outputs, make sure you get the one that's for the benchmarks, not the tests. You may also want |
102 | to `cargo clean` so you have only one `benchmarks-` binary (they tend to accumulate). | |
103 | ||
104 | ```bash | |
105 | sudo perf record target/release/deps/benchmarks-* --bench decode_10mib_reuse | |
106 | ``` | |
107 | ||
108 | Then analyze the results, again with perf: | |
109 | ||
110 | ```bash | |
111 | sudo perf annotate -l | |
112 | ``` | |
113 | ||
114 | You'll see a bunch of interleaved rust source and assembly like this. The section with `lib.rs:327` is telling us that | |
115 | 4.02% of samples saw the `movzbl` aka bit shift as the active instruction. However, this percentage is not as exact as | |
116 | it seems due to a phenomenon called *skid*. Basically, a consequence of how fancy modern CPUs are is that this sort of | |
117 | instruction profiling is inherently inaccurate, especially in branch-heavy code. | |
118 | ||
119 | ```text | |
120 | lib.rs:322 0.70 : 10698: mov %rdi,%rax | |
121 | 2.82 : 1069b: shr $0x38,%rax | |
122 | : if morsel == decode_tables::INVALID_VALUE { | |
123 | : bad_byte_index = input_index; | |
124 | : break; | |
125 | : }; | |
126 | : accum = (morsel as u64) << 58; | |
127 | lib.rs:327 4.02 : 1069f: movzbl (%r9,%rax,1),%r15d | |
128 | : // fast loop of 8 bytes at a time | |
129 | : while input_index < length_of_full_chunks { | |
130 | : let mut accum: u64; | |
131 | : | |
132 | : let input_chunk = BigEndian::read_u64(&input_bytes[input_index..(input_index + 8)]); | |
133 | : morsel = decode_table[(input_chunk >> 56) as usize]; | |
134 | lib.rs:322 3.68 : 106a4: cmp $0xff,%r15 | |
135 | : if morsel == decode_tables::INVALID_VALUE { | |
136 | 0.00 : 106ab: je 1090e <base64::decode_config_buf::hbf68a45fefa299c1+0x46e> | |
137 | ``` | |
138 | ||
139 | ## Fuzzing | |
140 | ||
141 | This uses [cargo-fuzz](https://github.com/rust-fuzz/cargo-fuzz). See `fuzz/fuzzers` for the available fuzzing scripts. | |
142 | To run, use an invocation like these: | |
143 | ||
144 | ```bash | |
145 | cargo +nightly fuzz run roundtrip | |
146 | cargo +nightly fuzz run roundtrip_no_pad | |
147 | cargo +nightly fuzz run roundtrip_random_config -- -max_len=10240 | |
148 | cargo +nightly fuzz run decode_random | |
149 | ``` | |
150 | ||
151 | ## License | |
152 | ||
153 | This project is dual-licensed under MIT and Apache 2.0. | |
154 |