The fact that you don't ever feel "good" is something that stands out to me.
I have gone the last 2 years without medication and I just kept slowly slipping downward, so that my irritable/fatigued time was increasing and my ok time was decreasing ever so slowly. I just went back on medication because I decided life is too short to just be getting by and not really enjoying anything.
I really encourage you to talk to your doctor and to be open to the idea of medication, or whatever he/she might suggest. I fought medication for the longest time so I know the feeling of not wanting to be on it.