Stephen Burch's Birding & Dragonfly Website
(now with EOS R5 update October 2020!)
Simply looking at digital camera pictures and trying to make valid assessments and comparisons of noise levels may have its place, but the results can be very subjective and depend hugely on the subject matter of the picture, as well as the camera used and all the settings involved in getting the image out of the camera onto the screen. The size the image is viewed at can also makes a big difference. An image at 20% magnification can appear virtually noise free, but show it at 1:1 and up comes the noise.
Hence, I have taken a different approach and developed a reasonably simple method for quantitatively measuring pixel to pixel noise levels on digital camera pictures, effectively analysed at 1:1 magnification. I've then applied this method to images taken with different Canon digital cameras at different ISO settings, with some interesting results (to me anyway!).
The cameras assessed to date are the EOS 350D, EOS 40D, EOS 50D, EOS 7D, EOS 7D Mk II and EOS R5.
In general, the main factors that affect the noise level on an digital camera image are as follows:
One of my first tasks was then to see how its image noise levels compared with my trusty 7D Mk II that I had been using for almost six years since I acquired it in December 2014 (see below). I was hopeful that almost 6 years of sensor development would have resulted in a tangible improvement in the image noise level compared with the 7D Mk II. From results given on the excellent DXOMark website for the Canon EOS 5DS, which has a similar sensor element size to the R5 (and the 7D Mk II), I was anticipating about a 1 stop or more improvement with the R5.
From the plot above, it is good to see that the noise levels from the EOS R5 show the hoped for improvement of about 1 stop compared with the older EOS 7D II, so that for example the noise levels for the R5 with an ISO of 3200 are very similar to those from the 7D II at ISO 1600. Indeed, the R5 improvement appears to be slightly more than one stop, so that if anything the corresponding 7D results for one stop lower ISO are slightly above those for the R5 at one stop higher.
The R5 has a very slightly larger sensor element size than the 7D Mk II but such a small margin it is unlikely to affect these results significantly, so it seems that the 6 years that have elapsed between the launches of these two cameras have produced about a factor two improvement in sensor sensitivity.
This is a great finding, and suggests this camera upgrade has produced a noticeable improvement in the maximum ISO that can be used without noticeable reduction in image quality. With the 7D II, after using NeatImage for noise reduction, I was generally OK with using ISO settings up to 3200, but I was reluctant to go beyond this (e.g. ISO 6400) except in special circumstances (e.g. trips to rainforests where light is always at a premium).
So these findings suggest I should be able to use the R5 with ISO settings up to about 6400 before the noise (after NeatImage) becomes noticeable. Experience with this new camera in the field over the coming weeks and months will tell me if this is correct!
It will also be interesting to see how these results for the R5 compare with those on the DXOMARK website when they eventually appear. This website has very useful plots of noise level for large numbers of cameras, expressed in terms of 18% signal-to-noise ratio. This site also has a very handy compare tool that allows the results for up to 3 different cameras (from any manufacturer) to be shown on the same plot.
Sony was previously the market leader on sensor technology, but perhaps Canon has at last made up significant ground in this key area, as it appears to have done with the remarkable animal eye detection auto-focus in the R5.
Note that, as seen before, the above curves show that the maximum noise standard deviation at each ISO setting increases by the square root of the ISO setting value for each camera (e.g. the noise at ISO 800 is square root of 2 = 1.4 larger than that at ISO 400, and the noise at ISO 1600 is 1.4 times larger than at ISO 800 and so on). This is exactly what would be expected if the main source of this noise is photon counting statistics - and, given the laws of physics, there is not much that can be done about that, apart from increasing the detection efficiency of the sensor.
ACR vs DPP4
The plot above shows that for the same ISO setting, ACR is apparently giving appreciably lower noise levels than DPP4 for the EOS R5. These differences amount to nearly half a stop I would estimate.
Hence for this reason, it seems I will be better using ACR instead of DPP4 for the images from the EOS R5, whereas the reverse was true for the 7D Mk II! All very confusing!
Colour issue and Raw Converter selection for
Hence to see if I could improve the image noise levels coming out of DPP4, I tried increasing the noise reduction parameters available. There are two - Reduce Luminance noise and Reduce Chrominance noise. These have values between zero (none) and 20. I tried a small value of 1 which is shown on the plot above for ISO 1600.
This gives a very similar maximum noise level (4.0) as ACR and better control of the noise in the lower grey level region (<50) (i.e. the shadows). However it is also necessary to consider how much the noise reduction is affecting the image sharpness - in general the higher the noise reduction the lower the image sharpness. Given this extra consideration I have now concluded it is best to use zero noise reduction in DPP4 and then to use NeatImage to do all the subsequent noise reduction.
Camera specific differences between DPP3 and DPP4
Canon EOS 7D Mk
First of all, see below for the noise level plot which shows results from the 7D Mk II compared with the 7D. Note that as previously, all noise reduction parameters were switched off both on the cameras and in Canon's Digital Photo Professional (DPP) conversion software for the raw images. As far as possible, this should give a 'level playing field' and allows assessment of the direct outputs from the camera sensors, without any post processing.
From the plot above, it is somewhat disappointing but not entirely unexpected to see that 7D Mk II is giving very similar noise levels to the 7D, for the same ISO setting and grey level values.
Only for intermediate grey levels (around 150) at ISO 400 and ISO 800 is there a slight improvement with the Mk II. For lower grey levels (between about 50 and 100) at ISO 1600, the Mk II actually appears slightly noisier than the 7D.
The 7D Mk II has however very slightly smaller pixels, so these results do at least show that generally the Mk II is performing slightly better than the 7D, taking this into account.
The above curves show that the maximum noise standard deviation at each ISO setting increases by the square root of the ISO setting value (e.g. the noise at ISO 800 is square root of 2 = 1.4 larger than that at ISO 400, and the noise at ISO 1600 is 1.4 times larger than at ISO 800 and so on). This is exactly what would be expected if the main source of this noise is photon counting statistics - and, given the laws of physics, there is not much that can be done about that, apart from increasing the detection efficiency of the sensor. That may however already be close to one.
For completeness, I also show below all the DPP results I now have available for the various Canon DSLR's I have owned since 2006!
The above plot certainly demonstrates a progressive improvement in sensor performance from the EOS 350D to the 7D, when the areas of the pixels are taken into account. However it would seem that this improvement has more or less stalled for the 7D Mk II.
See below for a description of the method used to generate these results, and some older plots obtained with Adobe Camera Raw (ACR) as well as DPP.
On the other cameras, I again made sure any similar options were off, although there are fewer available on the older models. Having done this, all the raw images should be as near as possible to what is coming off the sensor, with minimal processing. In other words a like for like comparison between the cameras, or 'level playing field'.
For these measurements I prepared and printed out a grey scale test chart - see below:
The purpose of this chart was to allow measurements of noise to be made for different image brightnesses from black through to white, to understand how the noise varied. The test chart was designed to have steps with different brightnesses, but each step was an even grey so that any variations in image brightness (or grey level) within the step would be due to noise and not the target.
To reduce any variations in image brightness due to printing imperfections in the test chart, I made sure all the images were slightly out of focus.
To process the raw files from each camera, I used either Adobe Camera Raw (ACR), as contained in various versions of PhotoShop Elements or the full Photoshop. In addition, I have used from time to time Canon's own raw converters, initially Digital Photo Professional version 3 (DPP3) and more recently version 4 (DPP4). In doing the conversion to tiff files (which are lossless, unlike jpegs), I made sure any processing parameters that might increase or decrease noise were off (e.g. sharpening, noise reduction).
I then used the ISee! software from BAM in Germany which is intended for the quantitative professional analysis of X-ray images. This is a download for the free version. Other free packages may also be available to do this analysis (e.g. Image J).
This software allows the user to interactively define a small analysis area, that I set at a constant size of 30 x 30 pixels in all images (more recently 50 x 50 pixels). The software gives a readout of the standard deviation of the image grey levels within this area, which is a good measure of the noise level. I then moved this area across each picture, and measured the noise levels for each step, taking an average of five readings in slightly different positions on each step. The averaging was done to improve the accuracy of the measurements.
The example below shows the analysis of an image using ISee! in progress. The procedure for the analysis is as follows:
Using the above procedure on different images, it is possible to build up a series of measurements that I could compare to study the variations in noise with various parameters, the main ones being image brightness or grey level, ISO setting, and of course camera type.
I should add that these measurements were made on a black & white versions of the tiff images (which is how they appear in the X-ray image analysis software), so that they presumably are measures of the luminance noise, and not chroma noise.
Results on older cameras (EOS 50D, 40D and 350D)
Also the graph shows that, as expected, increasing the ISO setting increases the overall noise levels, as expected. However there is a more subtle effect in that the variation in noise level with image brightness also changes with ISO setting and is much higher for ISO 1600 than ISO 400 (i.e. the noise in the darker areas of an ISO1600 shot is much worse than in the brighter areas, whereas at ISO400 the noise varies much less with image brightness).
The results below certainly lend support to the concept of keeping the exposure as high as possible to avoid under-exposed darker areas. This is apparently called ETTR (Expose To The Right), i.e. make sure your histogram of image values is over to the right (bright), not left (dark).
With raw files, I've also found that if the image is slightly clipped or saturated (spike at the right edge), then reducing the exposure in the raw converter gets over this problem, and the spike goes away. Very useful if you are trying to keep exposures up, and sometimes go too far!
With the 50D and 7D, there is about a full stop of leeway on the exposure adjustment. That is you can reduce the exposure in the raw converter by about a stop to avoid saturation. Beyond that, saturated areas just stay saturated. On the older 350D, there is somewhat less leeway.
350D, 40D and 50D noise levels compared (ACR)
The result for the 40D appears impressive, as it has pixels about 20% smaller in area than the 350D (10 Mpixels compared with 8Mpixels for the 350D). However, it is not quite as good as that, since the 40D is calibrated so that for a given ISO setting it is about 1/4 stop less sensitive than the 350D, so the overall improvement in noise is probably only about 1/4 stop.
The result for the 50D is poorer than I was expecting, and something of a surprise. Given it has almost twice as many pixels (15 Mpixels) in the same area of sensor as the 350D, the noise would be expected to increase. In fact given this increase in number of pixels by almost a factor of two, a one stop increase in noise levels is almost exactly what would be expected if all the noise were coming from photon counting statistics. But that is relative to the 350D. If you compare the 50D with the 40D, the comparison is less favourable. What happened to the claimed benefits of larger micro-lenses, increasing the effective area of each pixel on the 50D sensor?
Because of this surprising result, I looked to see what other measurements/comparisons of noise levels were available on the web. Perhaps there were errors in my results, or even my camera was a poor model. However, frustratingly there are very few direct comparisons to be found, and nearly always the exact details of the in-camera noise settings or post processing noise reduction settings aren't given, so it is impossible to know what is being compared with what. One exception to this is the excellent DP review website, which gives a true like for like comparison of raw images from the 40D and 50D, with no noise reduction in the raw conversion either. Sure enough the 50D images are noticeably noisier, but no attempt is made to measure the differences.
The site also make the following erudite comments:
"Removing any in-camera noise reduction and processing the images using Adobe Camera Raw (without NR) gives us the nearest thing to a 'level playing field' for assessing the relative noise levels of the two cameras' sensors. Despite the fact that the 50D is the newer camera it shows visibly more chroma and luminance noise than the 40D. Considering the 50D's much more tightly packed sensor (4.5 MP/cm² vs 3.1 MP/cm² on the 40D) this comes hardly as a surprise. It would have been unreasonable to expect Canon's engineers to overcome the laws of physics."
I'd agree with that, but
it would have been useful to know by what factor the
noise was higher on the 40D than the 50D, to see how it
compares with the results below. Visual comparisons of
their shots are difficult, but to my eyes the differences
could be as high as 1 to 1.5 stops.
Camera Raw (ACR) compared with Digital Photo Professional
(DPP) - on EOS50D
Out of interest, I thought I would try as an experiment using the raw converter in Canon's free Digital Photo Professional (DPP V3.6.1) instead of my usual Adobe Camera Raw (ACR) in PhotoShop Elements. Imagine my surprise when I found that DPP was giving lower noise levels in the darker areas of the images than ACR, even with all the noise reduction turned off. The graph below shows the results for the 50D at ISO 400 and ISO 800. The improvement obtained using DPP is quite marked. For example at ISO 800, the noise in the darkest parts of the image with DPP is less than the noise at ISO 400 with ACR!
I'm unclear of the reasons for this difference. It could of course be that some hidden noise reduction is going on in DPP, without the user knowing about it. Whatever the reason, it appears better to use DPP than ACR for 50D images, certainly at the higher ISO settings.
To show this makes a difference in
practice, have a look at the example below, which shows a
1:1 crop of a somewhat underexposed pic of a Yellow-legged
Gull, taken with the 50D at ISO 400. The chroma noise
from ACR 5.4 (top) is noticeably worse than with DPP 3.6.1
(bottom) - just look at the darker areas on the back and
wing tips. Even after using NeatImage (result not shown),
the DPP result is appreciably better - NeatImage in my
experience can struggle with chroma noise.
Here is another example, again 1:1 crops, with exposure +0.85 in both ACR and DPP. This was a distant bird, taken against the light, without the sun out, so the underparts were quite dark. Again to my eyes, the chroma noise is appreciably better in DPP (3.6.1) than ACR (5.4).
Of course, I don't have a side-by side comparison obtained with the 350D (or my now ex 40D), but in my experience this sort of 1:1 noise level with the 50D is more characteristic of that seen at ISO 800 with the 350D, which is in accord with the measurements presented above.
Adobe Camera Raw
(ACR) compared with Digital Photo Professional (DPP) - on
The graph below shows the DPP and ACR noise levels at ISO 400, 800 and 1600 for the 350D. As for the 50D, the DPP results show lower noise levels in the darker areas of the image (to the left), but look what happens on the right hand side. Unlike for the 50D, DPP for the 350D gives noticeably higher noise levels than ACR for the brighter areas of the image - almost 1 stop worse in the brightest areas. In fact, over most image grey levels, ACR gives significantly lower noise levels than DPP!
Hence for the 350D, it seems that ACR is preferable to DPP - the reverse of the situation for the 50D. Again I have no idea for the explanation for these differences!
EOS 7D compared with EOS 50D and
EOS 350D using Digital Photo Professional (DPP) New
From the plot below, you can see that the noise levels on the 7D are quite impressive. Despite having slightly smaller pixels than the 50D, the 7D is giving significantly lower noise, especially at ISO 1600. The difference is smaller at the lower ISO settings, but is still noticeable in the darker areas of the image. The prehistoric 350D is similar to the much newer 50D, but of course only has about half the number of pixels of the 50D. Hence the 7D may be performing at a similar noise level to the much older 40D (which of course only had 10 Mpixels)!
These measurements do tend to confirm other reports - that Canon have improved the noise position on the EOS 7D, compared with the 50D. This is some reassurance that the expensive upgrade to the 7D may have been worthwhile! Time will tell if I continue to believe that!
However, what really matters to me is the noise left after the noise reduction software I use on all images - NeatImage. Of course that is a whole new subject in its own right! All I will say on that subject at present is that I often found that quite noisy ISO 800 50D images cleaned up really well with NeatImage, provided they were well exposed (i.e. ETTR).
It is very early days yet, but I'm not convinced NeatImage is doing quite such a good job on the 7D images at ISO 800. These appear to have quite a number of "rogue" pixels, either singly or in small groups, which are largely unaffected by NeatImage. So could it be that after NeatImage, the 50D is "better" than the 7D, despite its higher raw noise levels? Surely not?!
But at ISO 1600, the 7D images appear to clean up quite well with NeatImage, and overall I'm pretty impressed with the results at this high ISO setting.