Outbound Shipping and Return Shipping Costs (Buyer Mistake)

Hey guys looking for some help or additional info here. I’m a third party seller and I ship all of my own items. I was under the impression that when a customer purchased something by mistake (or another reason not due to the seller or defect) that the return shipping label would be deducted from their refund. Also since I do not offer free returns I thought the outbound shipping costs would also not be refunded. Due to the items I sell I unfortunately receive a lot of returns from misplaced orders. After reviewing some of the returns I noticed that the customer was receiving a full refund (the total amount they paid in the first place) no matter what the reason was for the return. So I am paying outbound shipping, return shipping and in most cases I cannot resell the product so am also losing the inventory. With the amount of returns I am receiving I’m losing close to a grand each month from shipping costs and product loss. Does this sound accurate or am I missing something here? Anyway to try and recoup any of the shipping costs? Any help would be greatly appreciated! Thank you