I have the package 'libertine' installed, which provides TeX version of the Linux Libertine fonts. If I in addition have ttf version of the fonts installed in the system, compiling the above example with XeLaTeX results in garbage (collection of strange accented characters), without any errors or warnings from XeLaTeX. If I remove the TTF fonts, I get the text but without ligatures. (OTF version of the fonts do not have any effect.) What is going on?
Note 1: Changing the font name to {Linux Libertine O}, or replacing the line with "\usepackage{libertine}", fixes the problem. But I would still like to understand what is happening.
Note 2: The previous note is not valid if one installs the ttf fonts from the Debian/Ubuntu ttf-linux-libertine package - there, the ttf font's name is "Linux Libertine O", so the solution from above will result in garbage - while the original code (using "Linux Libertine") will still result in a text without ligatures. (This is reported as a bug in Debian, but has not been fixed yet.)
It must be a bug specific to Debian and Ubuntu, because on both of my Fedora systems (32bit F13, 64bit F14) running TeX Live 2010 your code (as is, without modification) seems to work fine when compiled with XeLaTeX; I get ligatures with the fi, fl, and double dash. In Fedora the "linux-libertine-fonts" package provides fonts named "Linux Libertine" and "Linux Libertine C", there isn't a "Linux Libertine O" as in your system.
Maybe you could try using the fonts from the Fedora package and see if that helps.
Michał did you find a solution for this issue? I have very same problem with \set*font option. I'm running both MiKTeX (standalone) and LiveTeX (under Cygwin).
Each time I use non-standard font eg. \setromanfont{Minion Pro} all shortcuts for dashes, quotes, etc. go away, hence: