Hi,
Currently i am not using sendmail. But for attachment size, following is the parameter to change...
define(`confMAX_MESSAGE_SIZE',maxsize)
Eg:
define(`confMAX_MESSAGE_SIZE',10485760)
(10Mb in bytes)
As you are are using roundcube, check your php.ini file for the following parameters.
upload_max_filesize
post_max_size
memory_limit
Test this in a test environment.
Regarding the quota, usrquota and grpquota is the only known option to me.