Graphics, Figures & Tables ⇒ Tikz for matlab plots - randomly fails

[jacql]

I apologise for the long essay, but it's an obscure error with no useful messages.

I'm writing up my thesis and want to include a lot of MATLAB plots. I'm using matlab2tikz to export the plots into tikz files and, for those which work, they look fabulous in the LaTeX result, so I'd really like to get them to work, if possible. I've used the same code to export 8 similar graphs and I've checked that the files are absolutely identical (other than the data, of course), but three of them refuse to work. I don't think it has anything to do with the tikz code in the files themselves, because I got one of them to work when I changed the order in my .tex file to run it first. But now there are three left which won't work, even if they are the only tikz input calls in the .tex file.

I've trawled through the logfiles produced for each image and can see no errors, other than a few of these which occur just as the image is being built:
Overfull \hbox (8.30591pt too wide) in paragraph at lines 23--23

In the Build output window, all that happens is this:
----------
"filename"This is pdfTeX, version 3.1415926-2.4-1.40.13 (MikTeX 2.9 64-bit)
entering extended mode
===== 'mode=convert with system call': Invoking 'pdflatex -shell-escape -halt-on-error -interaction=batchmode -jobname "
tikzfigures/xrdFitCa415" "\def\tikzexternalrealjob{Thesis}\input{Thesis}"' ========
)
<some Overfull hbox messages, because the image isn't large enough, I think>
[4 <xrdFitCa415.pdf>]
------------

The external call to pdflatex works and an image is produced, but it's only a fragment of the full plot and all squashed up (see

xrdFitCa415.pdf

failed pdf

(45.02 KiB) Downloaded 460 times

). It should look like this:

xrdFitCa355.pdf

this

(176.8 KiB) Downloaded 422 times

and there is no code difference between the tikz files.

Can it have something to do with the plot size (there are 6 datasets per figure)? I switched to externalization because I had memory errors before, but perhaps there is still some sort of limit to how much data can fit into the LaTeX result?

Alternately, can I convert each image to pdf manually from the tikz files and if so, how? Perhaps then I would see more errors.

Any help will be greatly appreciated. Also let me know if you need more information.

[localghost]

Without a concrete code example it is nearly impossible to spot the probable cause for the problem. Perhaps it's easier to export the data sets to CSV files and plot them with pgfplots. If you are unsure how to do that, attach a sample data set (e.g. that of the presented plot) to your next post by upload to the forum server and we will see what can be done. If there are functions to be plotted, please add them completely with all parameters.


Best regards and welcome to the board
Thorsten

[jacql]

Fair enough - I just wasn't sure which code bits to display and was hoping someone would recognise it as a similar problem.

In my main project .tex file, I have:

Code: Select all

%% Packages for Graphics & Figures %%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{graphicx} %%For loading graphic files
\usepackage{caption}
\usepackage[position=top]{subfig}
\usepackage{float}
\usepackage{tikz,pgfplots}
\pgfplotsset{compat=newest}
\pgfplotsset{
  every tick label/.style = {font=\scriptsize},
  every axis label/.style = {font=\scriptsize},
}
% needed for memory management with tikz images
\usetikzlibrary{external}
\tikzexternalize[optimize=false,prefix=tikzfigures/]
% define these for drawing figures
\newlength\figurewidth
\newlength\figureheight
\newlength\fullwidth
\newlength\fullheight
\setlength\fullwidth{14cm}
\setlength\fullheight{9cm}

\DeclareGraphicsExtensions{.pdf,.jpg,.png}

This file includes a .tex file, which further includes a .tex file - mentioning this, just in case it's a case of overnesting?. In this last .tex file is the code to call the figures:

Code: Select all

\begin{figure}[H]
\begin{center}
		\setlength\figurewidth{5.5cm}
		\setlength\figureheight{6cm}
		\subfloat[300 K]{
			\tikzsetnextfilename{xrdFitNaRT}
			\input{figures/xrdFitNaRT.tikz}
			\label{fig:xrdNaFitsRT}
		}
		\subfloat[355 K]{
			\tikzsetnextfilename{xrdFitNa355}
			\input{figures/xrdFitNa355.tikz}
			\label{fig:xrdNaFits355}
		}\qquad
		\subfloat[415 K]{
			\tikzsetnextfilename{xrdFitNa415}
			\input{figures/xrdFitNa415.tikz}
			\label{fig:xrdNaFits415}
		}
		\subfloat[475 K]{
			\tikzsetnextfilename{xrdFitNa475}
			\input{figures/xrdFitNa475.tikz}
			\label{fig:xrdNaFits475}
		}\qquad
 \caption{blah blah}
\label{fig:xrdNaFits}
\end{center}
\end{figure}

\begin{figure}[H]
\begin{center}
		\setlength\figurewidth{5.5cm}
		\setlength\figureheight{6cm}
      \subfloat[355 K]{
				\tikzsetnextfilename{xrdFitCa355}
				\input{figures/xrdFitCa355.tikz}
				\label{fig:xrdCaFits355}
			}
      \subfloat[415 K]{
				\tikzsetnextfilename{xrdFitCa415}
				\input{figures/xrdFitCa415.tikz}
				\label{fig:xrdCaFits415}
			}\qquad
      \subfloat[475 K]{
				\tikzsetnextfilename{xrdFitCa475}
				\input{figures/xrdFitCa475.tikz}
				\label{fig:xrdCaFits475}
			}
      \subfloat[300 K]{
				\tikzsetnextfilename{xrdFitCaRT}
				\input{figures/xrdFitCaRT.tikz}
				\label{fig:xrdCaFitsRT}
			}\qquad
    \caption{blah blah}
    \label{fig:xrdCaFits}
\end{center}
\end{figure}

[jacql]

Update: I have discovered that the small, squashed up image is due to not resetting the figure dimensions just before calling, however this still does not resolve the problem of some images just not forming at all.

When I isolate each tikz input into its own .tex file and include only this file (leaving out the full set of figures) then each independent image works. This means that I can get around the problem by creating each one separately and thereafter tikz sees that a .pdf exists and moves on. However, it would be nice to know why they don't work together. (this is why I suspected some sort of memory error, despite externalizing)

[localghost]

Please provide a self-contained and minimal example as described on the page when following the link in my last reply. At the moment your code is quite useless because not compilable as is. And we would appreciate it very much if you use the

Code: Select all

 button to tag code as such.

[jacql]

Unfortunately, it seems this problem is one where making a self-contained minimal example is not possible and also takes much more than just half an hour (I've just spent over 3 hours on it and now the .tex file is too big to attach). It is very elusive and I have encountered several red herrings along the way to working out what makes this problem "minimal".

The error only happens when I put all the figures in. Since it must be self-contained, this makes one very big file. I have 10 figures, all of which have 6 datasets within them. When I only add 8 of the figures, it works. When I go over this, it fails with this error:
! TeX capacity exceeded, sorry [buffer size=200000].

However, before assuming that it's just that my datasets are too big (which is probably also true), it's not quite as simple as that.
a) Even if the pdf files for the previous 8 files (which all worked) have been created in a previous build (i.e. the tikz skips them on subsequent builds), the ninth one fails with the same error, which does not make sense.
b) If I move the last figure up to the top, it works fine.
c) This buffer exceeded error is somewhat random: I've just added a tenth figure at the end and it works fine suddenly.

So my questions boil down to:
1. Has my externalization not worked properly?
2. Despite externalization, is there still some sort of limit to how big the plots can be? (which seems a bit odd: PhD theses are notoriously big on datasets and I've only put in 9 so far!)
3. If there is a limit, why is it so capricious?

If you still need the code before you'll consider my question, please can I submit the .tex and the figure .tikz files separately?

[localghost]

Unfortunately, it seems this problem is one where making a self-contained minimal example is not possible and also takes much more than just half an hour (I've just spent over 3 hours on it and now the .tex file is too big to attach). It is very elusive and I have encountered several red herrings along the way to working out what makes this problem "minimal". […]

But how are we supposed to help if the problem is not reproducible for others? Do you have an idea? I don't. You can choose between struggling endlessly with that problem or invest the time to cut it down and so allow us to help. And until now you did say nothing regarding the suggestions in my first reply. At the moment I see no possibility to help you.

[…] If you still need the code before you'll consider my question, please can I submit the .tex and the figure .tikz files separately?

You can do so by upload to the forum server or ask by PM for other contact information. But exported code is mostly quite bad and hard to debug. If you have the data sets for the plot you presented in your question, please attach them also for working out alternatives. And perhaps a corresponding Know How article can bring you closer to a solution.

[jacql]

If you mean the suggestion about converting it to CSV files for pgfplots, I admit that I wasn't sure how to respond to that. Unless I misunderstood what you're saying (which is always possible :) that sounds like it would simply bypass the problem, which is specific to tikz. But please feel free to correct me: I am just a newbie user of tikz and pgplots.

But how are we supposed to help if the problem is not reproducible for others? Do you have an idea? I don't.

If no-one has ever seen this problem before, then of course that's true and our only recourse is to reproduce it elsewhere. I was hoping that it was something that had been encountered before. However, I do think that it should still be possible to answer my general question about memory limitations without actually reproducing the problem locally. Is it known that Tex struggles to deal with too many datasets, even though it's externalized? Or should what I want to do be possible, regardless of my dataset size & number?

I have already cut the file down and made it self-contained: I am not avoiding doing that by any means. However, the file upload size limit is stopping me from sharing it with you. But your comment:

If you have the data sets for the plot you presented in your question, please attach them also for working out alternatives.

leads me to think that we may be talking about the same thing. My tikz files include all the datasets (they were generated by matlab2tikz, as I said) and this is what makes them so big. If you're saying that I can strip the datasets out into separate files and submit those separately along with the self-contained .tex file, then I'll be happy to do that.

[Johannes_B]

I guess you could just put them all in a zip-file and attach the whole bunch to your next post.
However, help with projects as big as yours seems to be is not guaranteed.

A few words to explain your problem: Matlab plots your datapoints. Consider the following: You have two datapairs (1,1) and (2,2). Matlab plots them with a line inbetween. The tikzexport then outputs \draw-commands not only for the datapairs, but also for the line inbetween, the x-axis with all the ticks and the y-axis with all the ticks. Your simple plot can have more than a thousand new datapairs to draw everything.
On the other hand, as Thorsten suggested, pgfplots uses TikZ internally, either computing everything every time or converting the plot once to pdf and including the pdf afterwards. This is called externalizing.

Just see for yourself:

Code: Select all

%\documentclass{standalone}
\documentclass{article}
\usepackage{pgfplots}
\begin{document}
\begin{tikzpicture}
\begin{axis}[xlabel={Number of Chickens},
ylabel={Output $\chi$}
]
\addplot coordinates {
(1,1)
(2,2)
};
\end{axis}
\end{tikzpicture}
\end{document} 

Exporting your data as csv (meaning a simple text-file) is quite easy, it is something like save('filename.dat', '<variableA>', '<variableB>', '-ASCII'). Pgfplots can also plot data by reading a file, no need to copy-paste any data.

Best regards
Johannes

[feuersaenger]

Hi Jacql,

I am author of pgfplots and have read about your problems.

Let me stress that there is no upper limit on the number of plots which can be included into a document. Take a look at the pgfplots manual: ~400 pages containing ~400 plots (at least). And it does not even make use of externalization (only for expensive pictures).

I still did not see the actual error message and I am a bid confused what failed at all. The only thing that I can see is that you observed memory issues. Have you tried to compile it by means of lualatex? It is better at memory management.

Since your setup is quite complex and it seems to be difficult to create a minimum working example, I would like to make the following suggestion: you could cram the smallest available document which exhibits the problems into one zip archive, send it to me by email, and I will see if I can reproduce and perhaps explain your issue. Make sure that it compiles and remove as many packages as possible from its preamble.

Regarding the suggestions of others who recommended to export your data to CSV and import that into pgfplots: you already did that. The script matlab2tikz is actually a sophisticated CSV export tool from matlab to pgfplots.

If you agree that you can send your code to me, you can find my email address in the pgfplots manual. I will treat your files as confidential and will only post the answer of my analysis here.

Note that I have time today (September 1) and then again next Thursday.

Kind regards

Christian