bpf: improve JEQ/JNE path walking

verifier knows how to trim paths that are known not to be
taken at run-time when register containing run-time constant
is compared with another constant.
It was done only for JEQ comparison.
Extend it to include JNE as well.
More cases can be added in the future.

                     before  after
bpf_lb-DLB_L3.o       2270    2051
bpf_lb-DLB_L4.o       3682    3287
bpf_lb-DUNKNOWN.o     1110    1080
bpf_lxc-DDROP_ALL.o   27876   24980
bpf_lxc-DUNKNOWN.o    38780   34308
bpf_netdev.o          16937   15404
bpf_overlay.o         7929    7191

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: John Fastabend <john.fastabend@gmail.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 46ff4e5b3fb7d5291afb33e6d8b676a99a55c796..afe9a1a0a5fefc121389e96f4f72d51a22477342 100644 (file)
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -2955,8 +2955,9 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
        if (BPF_SRC(insn->code) == BPF_K &&
            (opcode == BPF_JEQ || opcode == BPF_JNE) &&
            dst_reg->type == SCALAR_VALUE &&
-           tnum_equals_const(dst_reg->var_off, insn->imm)) {
-               if (opcode == BPF_JEQ) {
+           tnum_is_const(dst_reg->var_off)) {
+               if ((opcode == BPF_JEQ && dst_reg->var_off.value == insn->imm) ||
+                   (opcode == BPF_JNE && dst_reg->var_off.value != insn->imm)) {
                        /* if (imm == imm) goto pc+off;
                         * only follow the goto, ignore fall-through
                         */