The house was built, probably in the middle 1860's, by Heinrich Pantermuehl, who had come to Texas with his father and nine brothers and sisters about 1850 or 1852. As can be seen, it was built entirely of hand-faced limestone quarried on the property. Originally it comprised two rooms and a separated, rather large, stone kitchen in the rear, with a breeze way between the two structures. This was fairly common in these early German homesteads, and kept the heat and smoke of the cooking area out of the living quarters. As was quite usual, the house was of story-and-a-half co nstruction, with the attic used for storage, tobacco drying, and at times as sleeping quarters for the children. The only access to the attic, as is also usual, was by means of an outside stairway.
Heinrich Pantermuehl moved from Fischer's Store to Smithson's Valley in the middle 1860's, and may have at first lived in a small house on the property while building the bigger homestead. The foundations of several such little houses have been found on the various properties at one time owned by Ben Smithson. The foundation stones of the little house can still be seen located a few yards south of the front part of the house.
Heinrich Pantermuehl was the son of Joachim Pantermuehl, who had emigrated to Texas from Prussia about 1850 to 1852. He came alone with ten children and settled at Fischer's Store. His wife died shortly before the family departed Germany, and local tradition has it that his wife, just before her death, made Joachim promise that he would go through with their plan and take all the children to Texas, which he did. Joachim never remarried, and continued to live at Fischer's Store until his death.