esto means" this" which is used when the adjective is not clearly mentioned in the sentence and este 'this' this is used when the noun or adjective is mentioned in the sentence.
Mi pasaporte está en el hotel."

In this sentence, está is neither feminine nor masculine -- Spanish verbs aren't gendered.

You're combining three different things here. First, "está," a form of the verb "estar." Second, "esta" and "este," the feminine and masculine forms of the word for "this." Third, "es," a form of the verb "ser."

If you were to use either esta (no accent) or este, this example sentence would read, "My passport this in the hotel." Clearly this doesn't work -- you need a verb.