So basically in the manga the author uses "boku" and "ore" for the gems which is just a masculine version of "I". Sometimes they even to refer to each other as brothers. But in the English dub version of the anime she instructed the translator to just use gender neutral pronouns like they/them because of her intentions to portray the gems as genderless and not even having a concept of gender to begin with. The subs unfortunately don't do this with the exception of Dia who may use "watashi", which is a more feminine (but can also be gender neutral depending on context) version of "I". At first it can be a lot to take in but you put it best, they're literally rocks