It doesn't have to be useful.
How about a Rifles capbadge, lovingly embroidered and framed?
Hip flask is good (must be full of a spirit (rum, brandy or whiskey)) or under Queens Regs your son must refuse it and inform the RSM that the presenter of said gift is to be ragged around the square for half an hour.
Get a good, solid metal one, no cups or bits that might break. He'll love you for it when he needs a nip.
Mine is dented and scratched to buggery now, but I take it everywhere. I even had it on me when I got married (much to the bridesmaids mirth). I sometimes put lucazade sport in it, and drink that on the ranges!!
In these modern enlightened times where hip flasks might not be as tolerated, how about a good quality metal flask?
Again, shop about to find a good, sturdy flask, with no civvie extras like carryings straps and extra cups. The push button type are awesome as they keep warm longer. And you know he'll love you for it he is on stag at 2 in the morning, at -20!!!!!
To be honest though I would shy away from military kit. One he will buy what he needs (or ask you for it as a birthday/christmas present

) and there is the risk he will lose a torch or similar. (same with the flask to be honest).
The first idea, also sort of gives him the feeling that you are buying in to his new extended family. And that you support him.
How about a set of Rifles cuff links? or silk tie? If you think he will make a real career out of it, a pocket watch, Rifles capbadge engraved, and the date he passed out. Leave space for other auspicious dates (LCpl, Cpl and Sjt). He could wear his pocket watch when he gets to wear mess dress, a bit of a driver everytime he looks at it.