For reference, I add here what I did on my RHEL5 (Redhat Enterprise Linux) computer with TeX Live in order to increase memory:
$ cd ~
$ mkdir -p texmf/texmfcnf
$ cp /usr/share/texmf/web2c/texmf.cnf ~/texmf/texmfcnf/
$ export TEXMFCNF=~/texmf/texmfcnf/:
Now I edited my copied texmf.cnf, the line with main_memory got replaced by
main_memory = 7000000
Then I ran
$ fmtutil --all
which gave some errors, but apparently put some newly generated fmt-files in ~/.texmf-var. After this, pdflatex had more memory to cope with.
MiKTeX and proTeXt ⇒ Expanding memory
NEW: TikZ book now 40% off at Amazon.com for a short time.