swscale/ops_dispatch: make offset calculation code robust against overflow

As well as weird edge cases like trying to filter `monow` and pixels landing
in the middle of a byte. Realistically, this will never happen - we'd instead
pre-process it into something byte-aligned, and then dispatch a byte-aligned
filter on it.

However, I need to add a check for overflow in any case, so we might as well
add the alignment check at the same time. It's basically free.

Signed-off-by: Niklas Haas <git@haasn.dev>
This commit is contained in:
Niklas Haas
2026-04-15 20:32:18 +02:00
committed by Niklas Haas
parent 95e4f7cac5
commit 86307dad4a

View File

@@ -467,8 +467,17 @@ static int compile(SwsGraph *graph, const SwsOpList *ops, SwsPass *input,
goto fail;
}
for (int x = 0; x < filter->dst_size; x++)
offset[x] = filter->offsets[x] * p->pixel_bits_in >> 3;
for (int x = 0; x < filter->dst_size; x++) {
/* Sanity check; if the tap would land on a half-pixel, we cannot
* reasonably expect the implementation to know about this. Just
* error out in such (theoretical) cases. */
int64_t bits = (int64_t) filter->offsets[x] * p->pixel_bits_in;
if ((bits & 0x7) || (bits >> 3) > INT32_MAX) {
ret = AVERROR(EINVAL);
goto fail;
}
offset[x] = bits >> 3;
}
for (int x = filter->dst_size; x < pixels; x++)
offset[x] = offset[filter->dst_size - 1];
p->exec_base.in_offset_x = offset;