Code: Select all
\usepackage[nomath]{lmodern}
\big, \bigl, \bigr parenthesis), while it's perfect in 12pt (and also with 10pt, I think). It hasn't been updated after all these years? Or is there something wrong with using 11pt font size for a book? 
\documentclass[11pt]{article}
\usepackage[T1]{fontenc}
\usepackage{lmodern}
\begin{document}
\[(a) \big( a \big) \bigg( a \bigg) \Big( a \Big) \Bigg( a \Bigg)\]
\[\{a\} \big\{ a \big\} \bigg\{ a \bigg\} \Big\{ a \Big\} \Bigg\{ a \Bigg\}\]
\[[a] \big[ a \big] \bigg[ a \bigg] \Big[ a \Big] \Bigg[ a \Bigg]\]
\end{document}
\usepackage{lmodern} result in:
\usepackage[T1]{fontenc} and \usepackage{lmodern} it is:
nomath for lmodern does not change a lot, because it would simply use the European Computer Modern Math in 11pt. But if this is the wanted result, there is not really an argument against using this option. So just do it.\big( is larger than (. The documentation more or less says. that \Bigg > \bigg > \Big > \big. For example TeX by Topic explains:
The given pt-sizes are for a base size of 10pt (default of plainTeX). No word about a relation to the unsized delimiters.As an approximate measure, the Big delimiters are one and a half times as large
(11.5pt) as big delimiters; bigg ones are twice (14.5pt), and Bigg ones are two and
a half times as large (17.5pt).
( and the \big( etc. are the same size. So it is not characteristic for Latin Modern.\documentclass[10pt]{article}
\usepackage[T1]{fontenc}
\usepackage{lmodern}
\begin{document}
\[(a) \big( a \big) \bigg( a \bigg) \Big( a \Big) \Bigg( a \Bigg)\]
\[\{a\} \big\{ a \big\} \bigg\{ a \bigg\} \Big\{ a \Big\} \Bigg\{ a \Bigg\}\]
\[[a] \big[ a \big] \bigg[ a \bigg] \Big[ a \Big] \Bigg[ a \Bigg]\]
\end{document}
\big and \bigg you should use \Big that I've shown in the examples too.