A study of the views of all the Rishonim on the kezayis, as described in my monograph on the topic, shows very clearly that the Rishonim of Sefard held the kezayis to be the size of a regular olive, and the Rishonim of Ashkenaz only said differently because, as some of them explicitly admitted, they had never actually seen an olive.