You badge collecting or something? What units are you in?
And I don't think that special forces tend to go for Cadet Instructors who are clearly far too into it. If you were made an SMI at age 22 or whatever it was, then I would surmise that you were the golden boy of a senior officer in your local ACF, and then went straight to AI, and shot up the promotion ladder. I would also surmise that your lightning promotion has made you unpopular among other instructors.
Why not join a normal TA unit? It smacks of waltism to ask whether or not to join 21 SAS, HAC or 4 PARA. If you had said "I'm thinking of joining the TA, these units are in my local area, but theres also RE, R Sigs etc etc, then you'd probably get a bit more sympathy. Because my advice would be to join a normal TA unit first.