In 1927, Pierre de Coubertin was invited by the Greek government to Olympia, to attend the unveiling of a commemorative stele created in order to honour his actions to revive the Olympic Games. During his stay in Greece, he discussed the need for an academic centre for the study of the Olympic Movement and its trends with his friend Ioannis Chrysafis, who headed the Department of Physical Education at the University of Athens. Coubertin believed that the Olympic Movement should not deviate from its educational objectives and had written: "I have not been able to carry out to the end what I wanted to perfect. I believe that a centre of Olympic studies would aid the preservation and progress of my work more than anything else, and would keep it from the false paths which I fear".

Coubertin's ideas were in accord with the aims of the Hellenic Olympic Committee, which wanted to set up an academic centre modeled after the ancient Gymnasium, to lay down the foundations for the educational value of sport by carrying out studies and organizing classical games. The sudden deaths of Chrysafis (1930) and Coubertin (1937) prevented them from implementing their ideas.

One year after Coubertin's death and following his own wish, his heart was placed inside the commemorative stele in Ancient Olympia. This rekindled the idea for the establishment of a centre for the Olympic Games in Ancient Olympia. The project was taken up by Ioannis Ketseas, a student of Chrysafis at the National Gymnastics Club and Secretary of the Hellenic Olympic Committee and the German Carl Diem, a close associate of Coubertin devoted to the Olympic Movement and education. Diem and Ketseas, who had worked together for the first Torch Relay from Olympia to Berlin in 1936, decided to work toward the realization of a Centre of Olympic Studies. In 1938 they prepared a plan for an institution named International Olympic Academy and submitted it to the Hellenic Olympic Committee. The HOC adopted the plan, and that same year, incorporated in its Charter the establishment and operation of the IOA, as one of the Committee's aims.

At the 38th IOC Session in Cairo, the members of the IOC were informed by the HOC about the law recently adopted for the establishment of an Olympic Academy in Greece. One year later, in the 39th IOC Session in London, the IOC decided to place this foundation which would promote the Olympic ideals through education under its auspices.

Following the end of World War II, a detailed memorandum about the operation of the Academy, drafted by Ketseas and Diem, was submitted to the 41st IOC Session in Stockholm in June 1947. On the 28th of April 1949, the 44th IOC Session in Rome unanimously approved the establishment of the IOA, and assigned its implementation and operation to the Hellenic Olympic Committee, under the auspices of the IOC. Several years later and after a long struggle, the first IOA Session was scheduled for the summer of 1961 to coincide with the opening ceremony of the ancient stadium of Olympia, which had been excavated thanks to the initiative of Carl Diem, who ensured the expenses for the project.

The International Olympic Academy was officially inaugurated on the 16th of June 1961, and the proceedings of the first Session were headed by Cleanthis Palaeologos, Director of the Physical Education Department of the University of Athens, and the German Professor Lotz. From 1962 to 1990 the academic pursuits of the Academy were coordinated by its late Dean, Otto Szymiczek, whose contribution to the development of the Academy was invaluable; along with Professor Palaeologos, who had been made Honorary Vice-President of the Academy, they charted the course for three decades, and were both awarded the Gold Medal of the Academy.The contribution of N. Nissiotis, Professor of the philosophy of religion, was also instrumental in the scientific development of the IOA. He presided over the Academy from 1977 to his death in 1986.

During its first decade of operation, the activities of the IOA were limited to the International Session for Young Participants. From 1970, the IOA progressively implemented additional educational programmes devoted to the issues of the Olympic Movement. Today, many different events take place every year on the premises in Ancient Olympia. Until 1966, participants lived in tents, and the Sessions were held under the pine trees. The first buildings were completed in 1967, and they have gradually been added to with sports facilities and new buildings. The new conference centre was completed in 1994, bringing state of the art facilities to the service of the participants. In recognition of its contribution to the humanistic aim it serves and to the development of the Olympic Movement, the IOA was awarded the Bonacosa Award in 1961 and 1970, and the Olympic Cup in 1981. In 2021 the IOA premises were fully renovated with the financial support of the IOC.