Stephen Burch's Birding & Dragonfly Website
Simply looking at digital camera pictures and trying to make valid assessments and comparisons of noise levels may have its place, but the results can be very subjective and depend hugely on the subject matter of the picture, as well as the camera used and all the settings involved in getting the image out of the camera onto the screen. The size the image is viewed at can also makes a big difference. An image at 20% magnification can appear virtually noise free, but show it at 1:1 and up comes the noise.
Hence, I have taken a different approach and developed a reasonably simple method for quantitatively measuring pixel to pixel noise levels on digital camera pictures, effectively analysed at 1:1 magnification. I've then applied this method to images taken with different Canon DSLR cameras at different ISO settings, with some interesting results (to me anyway!).
The cameras assessed to date are the EOS 350D, EOS 40D, EOS 50D, EOS 7D and EOS 7D Mk II.
Latest results: Canon EOS 7D Mk
First of all, see below for the noise level plot which shows results from the 7D Mk II compared with the 7D. Note that as previously, all noise reduction parameters were switched off both on the cameras and in Canon's Digital Photo Professional (DPP) conversion software for the raw images. As far as possible, this should give a 'level playing field' and allows assessment of the direct outputs from the camera sensors, without any post processing.
From the plot above, it is somewhat disappointing but not entirely unexpected to see that 7D Mk II is giving very similar noise levels to the 7D, for the same ISO setting and grey level values.
Only for intermediate grey levels (around 150) at ISO 400 and ISO 800 is there a slight improvement with the Mk II. For lower grey levels (between about 50 and 100) at ISO 1600, the Mk II actually appears slightly noisier than the 7D.
The 7D Mk II has however very slightly smaller pixels, so these results do at least show that generally the Mk II is performing slightly better than the 7D, taking this into account.
The above curves show that the maximum noise standard deviation at each ISO setting increases by the square root of the ISO setting value (e.g. the noise at ISO 800 is square root of 2 = 1.4 larger than that at ISO 400, and the noise at ISO 1600 is 1.4 times larger than at ISO 800 and so on). This is exactly what would be expected if the main source of this noise is photon counting statistics - and, given the laws of physics, there is not much that can be done about that, apart from increasing the detection efficiency of the sensor. That may however already be close to one.
For completeness, I also show below all the DPP results I now have available for the various Canon DSLR's I have owned since 2006!
The above plot certainly demonstrates a progressive improvement in sensor performance from the EOS 350D to the 7D, when the areas of the pixels are taken into account. However it would seem that this improvement has more or less stalled for the 7D Mk II, probably because photon counting statistics are now the major source of noise in images from the 7D and 7D Mk II, and not much more can be done to improve the sensor photon detection efficiency.
See below for a description of the method used to generate these results, and some older plots obtained with Adobe Camera Raw (ACR) as well as DPP.
On the other cameras, I again made sure any similar options were off, although there are fewer available on the older models. Having done this, all the raw images should be as near as possible to what is coming off the sensor, with minimal processing. In other words a like for like comparison between the cameras, or 'level playing field'.
For these measurements I prepared and printed out a grey scale test chart - see below:
The purpose of this chart was to allow measurements of noise to be made for different image brightnesses from black through to white, to understand how the noise varied. The test chart was designed to have steps with different brightnesses, but each step was an even grey so that any variations in image brightness (or grey level) within the step would be due to noise and not the target.
To reduce any variations in image brightness due to printing imperfections in the test chart, I made sure all the images were slightly out of focus.
To process the raw files from each camera, I initially used Adobe Camera Raw, as downloaded for running in PhotoShop Elements versions 5 or 6. For the EOS 50D images, I used ACR 5.4, which is the latest version at the time of writing, together with PSE 6. For the EOS 350D and 40D images, which were done at the start of the 2009, I used an earlier version of ACR (4.5) with PSE 5. A quick check has shown no significant differences going back to the older EOS 350D and 40D shots and converting with ACR 5.4 in PSE 6.
In doing the conversion to 8-bit tiff files, I made sure any processing parameters that might increase or decrease noise were off (e.g. sharpening, noise reduction). I then used some freely downloadable ISee! software from BAM which is intended for the quantitative professional analysis of X-ray images. Other free packages may also be available to do this analysis.
This software allows the user to interactively define a small analysis area, that I set at a constant size of 30 x 30 pixels in all images. The software gives a readout of the standard deviation of the image grey levels within this area, which is a good measure of the noise level. I then moved this area across each picture, and measured the noise levels for each step, taking an average of five readings in slightly different positions on each step. The averaging was done to improve the accuracy of the measurements.
The example below shows the analysis of an image using ISee! in progress. The procedure for the analysis is as follows:
Using the above procedure on different images, it is possible to build up a series of measurements that I could compare to study the variations in noise with various parameters, the main ones being image brightness or grey level, ISO setting, and of course camera type.
I should add that these measurements were made on a black & white versions of the tiff images (which is how they appear in the X-ray image analysis software), so that they presumably are measures of the luminance noise, and not chroma noise.
Also the graph shows that, as expected, increasing the ISO setting increases the overall noise levels, as expected. However there is a more subtle effect in that the variation in noise level with image brightness also changes with ISO setting and is much higher for ISO 1600 than ISO 400 (i.e. the noise in the darker areas of an ISO1600 shot is much worse than in the brighter areas, whereas at ISO400 the noise varies much less with image brightness).
The results below certainly lend support to the concept of keeping the exposure as high as possible to avoid under-exposed darker areas. This is apparently called ETTR (Expose To The Right), i.e. make sure your histogram of image values is over to the right (bright), not left (dark).
With raw files, I've also found that if the image is slightly clipped or saturated (spike at the right edge), then reducing the exposure in the raw converter gets over this problem, and the spike goes away. Very useful if you are trying to keep exposures up, and sometimes go too far!
With the 50D and 7D, there is about a full stop of leeway on the exposure adjustment. That is you can reduce the exposure in the raw converter by about a stop to avoid saturation. Beyond that, saturated areas just stay saturated. On the older 350D, there is somewhat less leeway.
350D, 40D and 50D noise levels compared (ACR)
The result for the 40D appears impressive, as it has pixels about 20% smaller in area than the 350D (10 Mpixels compared with 8Mpixels for the 350D). However, it is not quite as good as that, since the 40D is calibrated so that for a given ISO setting it is about 1/4 stop less sensitive than the 350D, so the overall improvement in noise is probably only about 1/4 stop.
The result for the 50D is poorer than I was expecting, and something of a surprise. Given it has almost twice as many pixels (15 Mpixels) in the same area of sensor as the 350D, the noise would be expected to increase. In fact given this increase in number of pixels by almost a factor of two, a one stop increase in noise levels is almost exactly what would be expected if all the noise were coming from photon counting statistics. But that is relative to the 350D. If you compare the 50D with the 40D, the comparison is less favourable. What happened to the claimed benefits of larger micro-lenses, increasing the effective area of each pixel on the 50D sensor?
Because of this suprising result, I looked to see what other measurements/comparisons of noise levels were available on the web. Perhaps there were errors in my results, or even my camera was a poor model. However, frustratingly there are very few direct comparisons to be found, and nearly always the exact details of the in-camera noise settings or post processing noise reduction settings aren't given, so it is impossible to know what is being compared with what. One exception to this is the excellent DP review website, which gives a true like for like comparison of raw images from the 40D and 50D, with no noise reduction in the raw conversion either. Sure enought the 50D images are noticeably noisier, but no attempt is made to measure the differences.
The site also make the following erudite comments:
"Removing any in-camera noise reduction and processing the images using Adobe Camera Raw (without NR) gives us the nearest thing to a 'level playing field' for assessing the relative noise levels of the two cameras' sensors. Despite the fact that the 50D is the newer camera it shows visibly more chroma and luminance noise than the 40D. Considering the 50D's much more tightly packed sensor (4.5 MP/cm² vs 3.1 MP/cm² on the 40D) this comes hardly as a surprise. It would have been unreasonable to expect Canon's engineers to overcome the laws of physics."
I'd agree with that, but
it would have been useful to know by what factor the
noise was higher on the 40D than the 50D, to see how it
compares with the results below. Visual comparisons of
their shots are difficult, but to my eyes the differences
could be as high as 1 to 1.5 stops.
Camera Raw (ACR) compared with Digital Photo Professional
(DPP) - on EOS50D
Out of interest, I thought I would try as an experiment using the raw converter in Canon's free Digital Photo Professional (DPP V3.6.1) instead of my usual Adobe Camera Raw (ACR) in PhotoShop Elements. Imagine my surprise when I found that DPP was giving lower noise levels in the darker areas of the images than ACR, even with all the noise reduction turned off. The graph below shows the results for the 50D at ISO 400 and ISO 800. The improvement obtained using DPP is quite marked. For example at ISO 800, the noise in the darkest parts of the image with DPP is less than the noise at ISO 400 with ACR!
I'm unclear of the reasons for this difference. It could of course be that some hidden noise reduction is going on in DPP, without the user knowing about it. Whatever the reason, it appears better to use DPP than ACR for 50D images, certainly at the higher ISO settings.
To show this makes a difference in
practice, have a look at the example below, which shows a
1:1 crop of a somewhat underexposed pic of a Yellow-legged
Gull, taken with the 50D at ISO 400. The chroma noise
from ACR 5.4 (top) is noticeably worse than with DPP 3.6.1
(bottom) - just look at the darker areas on the back and
wing tips. Even after using NeatImage (result not shown),
the DPP result is appreciably better - NeatImage in my
experience can struggle with chroma noise.
Here is another example, again 1:1 crops, with exposure +0.85 in both ACR and DPP. This was a distant bird, taken against the light, without the sun out, so the underparts were quite dark. Again to my eyes, the chroma noise is appreciably better in DPP (3.6.1) than ACR (5.4).
Of course, I don't have a side-by side comparison obtained with the 350D (or my now ex 40D), but in my experience this sort of 1:1 noise level with the 50D is more characteristic of that seen at ISO 800 with the 350D, which is in accord with the measurements presented above.
Adobe Camera Raw
(ACR) compared with Digital Photo Professional (DPP) - on
The graph below shows the DPP and ACR noise levels at ISO 400, 800 and 1600 for the 350D. As for the 50D, the DPP results show lower noise levels in the darker areas of the image (to the left), but look what happens on the right hand side. Unlike for the 50D, DPP for the 350D gives noticeably higher noise levels than ACR for the brighter areas of the image - almost 1 stop worse in the brightest areas. In fact, over most image grey levels, ACR gives significantly lower noise levels than DPP!
Hence for the 350D, it seems that ACR is preferable to DPP - the reverse of the situation for the 50D. Again I have no idea for the explanation for these differences!
EOS 7D compared with EOS 50D and
EOS 350D using Digital Photo Professional (DPP) New
From the plot below, you can see that the noise levels on the 7D are quite impressive. Despite having slightly smaller pixels than the 50D, the 7D is giving significantly lower noise, especially at ISO 1600. The difference is smaller at the lower ISO settings, but is still noticeable in the darker areas of the image. The prehistoric 350D is similar to the much newer 50D, but of course only has about half the number of pixels of the 50D. Hence the 7D may be performing at a similar noise level to the much older 40D (which of course only had 10 Mpixels)!
These measurements do tend to confirm other reports - that Canon have improved the noise position on the EOS 7D, compared with the 50D. This is some reassurance that the expensive upgrade to the 7D may have been worthwhile! Time will tell if I continue to believe that!
However, what really matters to me is the noise left after the noise reduction software I use on all images - NeatImage. Of course that is a whole new subject in its own right! All I will say on that subject at present is that I often found that quite noisy ISO 800 50D images cleaned up really well with NeatImage, provided they were well exposed (i.e. ETTR).
It is very early days yet, but I'm not convinced NeatImage is doing quite such a good job on the 7D images at ISO 800. These appear to have quite a number of "rogue" pixels, either singly or in small groups, which are largely unaffected by NeatImage. So could it be that after NeatImage, the 50D is "better" than the 7D, despite its higher raw noise levels? Surely not?!
But at ISO 1600, the 7D images appear to clean up quite well with NeatImage, and overall I'm pretty impressed with the results at this high ISO setting.