The University Observatory, also known as the Department of Astronomy, was designed by architect Charles Barry. Located in the University Parks, it was built in 1873-1975 to house a school of astronomical physics, the main focus of which would be research. Howard Grubb, astronomical instrument maker, was commissioned by the University to build a 12.25 inch equatorial refracting telescope for the Observatory and Warren De La Rue, a wealthy amateur astronomer, donated a reflecting telescope. A lecture theatre was added to the Observatory in 1877-1878. Although Oxford was already home to the Radcliffe Observatory, built in the 1770s and owned by the Radcliffe Trustees, the University Observatory was the University's first own observatory. Its conception reflected the growing prominence of physics and the sciences more generally at the University; the Clarendon Laboratory had been constructed in 1868 and in 1870 Preliminary Honour Examinations had been introduced in Mechanics and Physics.

The construction of the University Observatory was overseen by the Savilian Professor of Astronomy, Charles Pritchard, who, along with John Phillips, the Keeper of the University Museum until 1874, helped to ensure that the proposal for a new observatory passed through Convocation. Holders of the Savilian Professorship, founded in 1619 by an endowment by Sir Henry Savile and attached to a fellowship at New College from 1877, were in charge of the University Observatory. At that time the Professor was elected by a board consisting of the Chancellor of the University, the President of the Royal Society, the Astronomer Royal, the Radcliffe Observer, the Warden of New College, an individual nominated by the Warden and Fellows of New College and another nominated by Hebdomadal Council.

The University Observatory was under the superintendence of the Visitors of the University Observatory, a body created by University Statute in 1875. The Visitors, meeting at least once a term, included the Vice-Chancellor, the Proctors, the Astronomer Royal, the Director of the University Observatory at Cambridge, the Radcliffe Observer and four others elected by Congregation. Each year the Savilian Professor of Astronomy presented a report to the Visitors of the University Observatory.

As Professor, Pritchard focused the work of the Department upon visual stellar photometry and the application of photography to the determination of stellar parallaxes. Under Herbert Hall Turner, Pritchard's successor, initially the work of the Observatory centred upon the Oxford zone of the Astrographic Catalogue, completed in 1905. However, technological advancements and a lack of investment during the First World War meant much of the astronomical equipment at the Observatory, unchanged since 1890, soon became obsolete. As a result, Turner later turned his attention away from astronomy, focusing instead upon geophysics and seismology; work for the International Seismological Summary took place at the Observatory 1920-1946 and a small new wing, devoted primarily to seismology, was added to the Observatory in 1928.

In 1930 a Committee on the Future of Astronomy in Oxford was formed in response to the Radcliffe Observatory's proposed move to South Africa, producing a report two years later. It was in response to this debate that Savilian Professor Harry Hemley Plaskett proposed additional funding for the University Observatory. Thus, a new solar telescope and spectroscope was constructed for the University Observatory in 1935 and the Observatory soon became a centre for the study of solar physics. Plaskett prioritised graduate research, recruiting graduate assistants to work at the Observatory. Another, larger, solar telescope was completed in 1954 and a small tower for the telescope and a tunnel for its spectroscope were erected to the north of the Observatory buildings in 1957.

The Observatory was renamed the Department of Astrophysics in 1961 and, in the decade that followed, the theoretical section of the department grew rapidly. An Interdepartmental Committee for Theoretical Astrophysics was established in 1974. The cluster of buildings known as the Science Area developed around the Observatory, hindering observational work and making the redevelopment of the Observatory site more appealing. Discussions took place regarding the possible move of the Observatory elsewhere, for instance to Heythrop College, but these plans failed to materialise.

In 1973 the Board of the Faculty of Physical Sciences recommended that observational work cease to be carried out at the Observatory from 1979, and that the department integrate more closely with other physics departments. In 1977 the Visitors of the University Observatory were abolished, having held their final meeting the previous year, and in 1990 a single Department of Physics was formed from Astrophysics; Atmospheric, Oceanic and Planetary Physics; the Clarendon Laboratory; Nuclear Physics; and Theoretical Physics.

Savilian Professors of Astronomy from 1870: