Yes, you can install OpenWRT or OpnSense on a Raspberry Pi. If you don't want to replace your current OS on the Pi, you'll have to manually work with iptables (if you use linux) or pf (if you use BSD). Schematically, your old/current situation:
┌─────────────┐ ┌──────────┐ │ │ │ │ │ LAN / Modem ├─────────┤ Computer │ │ │ │ │ └─────────────┘ └──────────┘ ┌─────────────┐ ┌──────────┐ │ │ │ │ │ LAN / Modem ├─────────┤ Firewall │ │ │ │ │ └─────────────┘ └─────┬────┘ │ │ │ ┌─────┴────┐ │ │ │ Computer │ │ │ └──────────┘(slight repetition:) the reason you'd use a firewall external to your PC is that it makes it impossible for any application to 'bypass' it since it doesn't actually know it is there. There are of course no guarantees as no software is perfect and firewalls are software too. But it's a whole lot closer to actually doing what you want it to than some random desktop application.
Technically you could also add a second interface on your computer and give it the option to completely bypass the firewall but that only helps if you can restrict your application of choice to only go out the firewalled interface.
In the new example, the Firewall has two network interfaces, but technically it can be done with only 1 interface if you have a network switch with VLAN support. For a Raspberry Pi you'd need a USB-Ethernet adapter for that second interface.
For OpnSense there are many examples; i.e. last year's release on a Pi 3: https://forum.opnsense.org/index.php?topic=14875.60
You can also run the latest release and a different Pi, i.e. a 4B. NanoPi works too.
OpenWRT has manuals too: https://openwrt.org/toh/raspberry_pi_foundation/raspberry_pi
