Begun in 1455, the building was one of the first Renaissance buildings in Rome; although it was built around the medieval tower at the right of its facade. Much of its stone was quarried from the nearby Colosseum, a common practice in Rome until recent centuries. It was originally inhabited by the Venetian cardinal, Pietro Barbo, who later became Pope Paul II. He supposedly had it built on this location in order to view the horse races in the adjacent Via del Corso.
The building became a papal residence, and Pope Pius IV gave use of much the building to the Republic of Venice for use as its embassy. Throughout the nineteenth century, the building was the seat for the Austrian ambassador to the Vatican.
In 1917, Italy regained possession of the building, and it was restored. Mussolini had his office in the Sala del Mappamondo, and used a balcony in the palazzo for many of his most notable speeches to people gathered in the Piazza Venezia below.
The Museo del Palazzo di Venezia is within the building. It contains galleries of art, predominantly pottery, tapestry, statuary from the early Christian era up to early Renaissance.