Joe was openly lesbian, usually dressed as a man, and had tattooed arms. She had numerous relationships throughout her life, including with Greta Garbo, Tallulah Bankhead, and Marlene Dietrich.
Carstairs was given a doll by her girlfriend, Ruth Baldwin, and named it Lord Tod Wadley, which she took everywhere, except into her speedboats, out of fear of losing it. Following her death in 1993, Carstairs was cremated along with the doll, and were buried in Sag Harbor, NY.
After selling her Bahamaian islands, Carstairs owned this home in the early 80s.