I know some people have made it work, my problem is that because I always use Lyx I'm not quite sure how to implement that solution. Let's forget about Lyx for the moment, how would you use that advice. I have to use bibtex because of a custom bibliography style, but forget about the custom style for this question.
The file 88591sca.csf mentioned in that link seems to be part of the package bibtex8bit which Miktex has already installed, I can locate that .csf file on my hard drive so I know I already have it and the ordering there specified is correct as far as I can tell. So I have my "sortingtest.bib" and "testingsorting.tex" files ready in the same folder, on the command line I navigate to that folder and then I type:
bibtex8 testingsorting.aux --csfile 88591sca.csf
These do produce a .pdf but the bibliography entries are not sorted correctly like the .csf file is supposed to specify. After the second command it reports:
The 8-bit codepage and sorting file: 88591sca.csf The top-level auxiliary file: testingsorting.aux The style file: apalike.bst Database file #1: sortingtest.bib
So it looks as if it should work (?). Basically is there something I'm doing obviously wrong in the above description. I can give you examples of the .tex and .bib files I use for testing: