tefilin - Is there an issue of arrogance to wear Tfilin Rabenu Tam

I have been putting on tefilin shel Rashi for many years and a friend recently said I should start putting on Tefilin Rabenu Tam as well. The Kitzur Shulchan Aruch says that only the especially pious put on tefilin shel Rabenu Tam and one should be careful not to appear arrogant ("tzidkut yeter")

Does the upside of purchasing and wearing tefilin shel Rabenu Tam outweigh the risk of appearing arrogant (and also having to explain to people that I am still a Litvak and not a Chabadnik...)