First, and likely foremost, the data produced from XML serialization is significantly larger than that from the binary serializer
Second: XmlSerializer uses reflection to access the public properties on your types. Your properties might not be simple get/set. In contrast, the BinaryFormatter looks directly at fields, so you are not affected by any less-than-efficient properties you might be serializing.
Note that both the XmlSerializer and BinaryFormatter provide a mechanism whereby you – as the author of the serialized types – can control serialization. See IXmlSerializable and ISerializable. AFAIK, only BinaryFormatter allows you to customize serialization when you’re not the author of the types being serialized (see ISerializationSurrogate).
Actually it is the BinaryFormatter that uses reflection and XmlSerializer generates code to Serialize and Deserialize and hence getting and setting values is actually faster in XmlSerializer (unless your property getter and setter does non trivial tasks)
However BinaryFormatter is usually found to be faster in cases where the serialized data is predominantly numeric (array of ints etc.) This is due to the fact that the XmlSerializer has to convert the data to/from string. XmlSerializer is also found to be slower than BinaryFormatter in cases where the objects are deeply nested (e.g. A contains B, B contains C, C contains D and so on)
XmlSerializer is significantly slower the first time since it works by generating helper classes to serialize and deserialize and the generated serializer has to be compiled. The subsequent invocation should be fast. To avoid the first time slow down one can use sgen.exe. You can find more information about this at http://msdn2.microsoft.com/en-us/library/bk3w6240(vs.80).aspx
If performance is the main reason for choosing the serialization technology then I strongly suggest you to look at the DataContractSerializer that ships in .NET 3.0 (as part of WCF). When used with the binary XML (also shipped as part of WCF) the DataContractSerializer outperforms both serializers in most of the scenarios. You can find more information about the DataContractSerializer at http://msdn2.microsoft.com/en-us/library/system.runtime.serialization.datacontractserializer.aspx. You can find info about the Binary XML at http://blogs.msdn.com/drnick/archive/2007/04/09/what-a-binary-encoding-means.aspx and http://www.wintellect.com/cs/blogs/jsmith/archive/2006/03/28/wcf-xmldictionaries.aspx