James Castle was born in 1899 to Frank and Mary Castle in the sparsely-populated mountain community of Garden Valley, Idaho. His parents served as postmasters in Garden Valley from 1901 to 1923 working out of the living room of the family home. James, the fifth of seven surviving children, was deaf from birth. He began drawing at about age six and from age ten to fifteen attended the School for the Deaf and the Blind in the Idaho town of Gooding, 150 miles away. The Castle family moved to Star, Idaho in 1923 and purchased the Star Grinding Mill. Frank Castle died in 1927 and Mary moved with James in 1931 to a property on the outskirts of Boise. When his mother died in 1948, James continued to live in the Boise house with his sister Agnes (Peggy), her husband Guy Wade and their four children until his death in 1977.
Castle had gained recognition as an artist in the early 1950s when his nephew Bob Beach, while attending the Museum Art School in Portland, Oregon, showed his uncle’s drawings to his instructors. Castle’s work was then included in exhibitions throughout the northwest with one-person shows in 1963 and 1976 at the Boise Gallery of Art, now the Boise Art Museum.