In 1600, merchant Martin Peller acquired a property on the Egidienberg from the Grolands, a family of Nuremberg patricians. He had the existing structure pulled down, and in 1602 began raising an impressive residential and commercial edifice in its place. Disputes with neighbors and the city council delayed the completion of construction until 1607. Peller, who lived at his father-in-law Bartholomäus Viatis's home at the Barfüsserbrücke until Viatis died, did not move into the new house until 1625, and only lived there for a few years, until his own death in 1629. The splendid building remained in the Peller family's hands until 1828, after which it passed through several other owners until it was acquired by furniture maker Johann Adam Eysser.
The City of Nuremberg bought the house from Eysser's descendants in 1929, as a location for the City Archives. By that time, the Pellerhaus already counted as one of Nuremberg's most beautiful attractions, and its magnificent interior courtyard had even earned it a reputation as one of the most significant structural monuments in the history of German art.
Apart from the cellar, a few parts of the ground floor and the staircase tower, the Pellerhaus was almost entirely destroyed during the air raids of World War II. The high cost of rebuilding in full led that option to be abandoned at an early stage; only the arched walls were preserved, up to and including the first floor above the ground. In 1957, these were incorporated into Fritz Mayer's new structure, planned as a library; it served multiple functions over the subsequent decades, including as the City Archive (until 2000) and as a branch of the City Library (until 2012). The Pellerhaus became a protected historical monument in 1998.
Ever since the City Library moved out in 2013, the building has been home to the German Games Archive in Nuremberg. In the course of this temporary use, the Association of the Nuremberg Municipal Museums took over managing the building, which also houses other temporary tenants in addition to the Archive. The next step for the Pellerhaus is now the planning for the new "House of Games" as part of the city’s proposal to be a Capital of Culture. From 2025 on, the restored and renovated building should then become one of the city’s central cultural locations and meeting places.