You can use rmmod command to remove module or device driver
Code:
lsmod
rmmod {module-name}
More about lsmod command ->
How to display or show information about a Linux Kernel module or drivers
More about rmmod command ->
Howto: Linux add remove a Linux kernel modules / drivers
Another option is to modify / edit file called /etc/modprobe.conf or /etc/modprobe.d/blacklist (Debian/Ubuntu) and append module name. This will avoid loading a Linux driver module automatically, see my tutorial ->
Howto: Prevent a Linux kernel module from auto loading for more info.
HTH