Stephen Burch's Birding & Dragonfly Website

Home | Trip Reports | Gallery | UK index | Oxon pics | UK pics | Dragonflies | Other Nature | Links


Quantitative measurements of DLSR raw image noise levels

Introduction
For some time now, I've been interested in the noise that appears on DSLR images, and the best methods for getting rid of it (see my comments about NeatImage for example on another page on this website). However, on the principle that knowing as much as possible about one's enemy is a good idea, I thought I'd do a bit more investigation of the subject.

Simply looking at digital camera pictures and trying to make valid assessments and comparisons of noise levels may have its place, but the results can be very subjective and depend hugely on the subject matter of the picture, as well as the camera used and all the settings involved in getting the image out of the camera onto the screen. The size the image is viewed at can also makes a big difference. An image at 20% magnification can appear virtually noise free, but show it at 1:1 and up comes the noise.

Hence, I have taken a different approach and developed a reasonably simple method for quantitatively measuring pixel to pixel noise levels on digital camera pictures, effectively analysed at 1:1 magnification. I've then applied this method to images taken with different Canon DSLR cameras at different ISO settings, with some interesting results (to me anyway!).

The cameras assessed to date are the EOS 350D, EOS 40D, EOS 50D, EOS 7D and EOS 7D Mk II.

Latest results: Canon EOS 7D Mk II
With my recent (December 2014) acquisition of an EOS 7D Mk II, I was of course eager to see how its noise levels compared with my original 7D and the other older cameras.

First of all, see below for the noise level plot which shows results from the 7D Mk II compared with the 7D. Note that as previously, all noise reduction parameters were switched off both on the cameras and in Canon's Digital Photo Professional (DPP) conversion software for the raw images. As far as possible, this should give a 'level playing field' and allows assessment of the direct outputs from the camera sensors, without any post processing. 

Canon EOS 7D Mk II noise levels

Noise level measurements for the 7D Mk II compared with the 7D as a function of image brightness or grey level for three different ISO settings (400, 800 and 1600). Also shown are the ISO 3200 results for the Mk II.

From the plot above, it is somewhat disappointing but not entirely unexpected to see that 7D Mk II is giving very similar noise levels to the 7D, for the same ISO setting and grey level values.

Only for intermediate grey levels (around 150) at ISO 400 and ISO 800 is there a slight improvement with the Mk II. For lower grey levels (between about 50 and 100) at ISO 1600, the Mk II actually appears slightly noisier than the 7D.

The 7D Mk II has however very slightly smaller pixels, so these results do at least show that generally the Mk II is performing slightly better than the 7D, taking this into account.

The above curves show that the maximum noise standard deviation at each ISO setting increases by the square root of the ISO setting value (e.g. the noise at ISO 800 is square root of 2 = 1.4 larger than that at ISO 400, and the noise at ISO 1600 is 1.4 times larger than at ISO 800 and so on). This is exactly what would be expected if the main source of this noise is photon counting statistics - and, given the laws of physics, there is not much that can be done about that, apart from increasing the detection efficiency of the sensor. That may however already be close to one.

For completeness, I also show below all the DPP results I now have available for the various Canon DSLR's I have owned since 2006!

DPP noise levels for all Canon DSLR cameras

All available noise level measurements for Canon DSLR's as a function of image brightness or grey level for three different ISO settings (400, 800 and 1600), using DPP for the raw image conversion. The ISO 3200 results for the 7D Mk II are also given.

The above plot certainly demonstrates a progressive improvement in sensor performance from the EOS 350D to the 7D, when the areas of the pixels are taken into account. However it would seem that this improvement has more or less stalled for the 7D Mk II, probably because photon counting statistics are now the major source of noise in images from the 7D and 7D Mk II, and not much more can be done to improve the sensor photon detection efficiency.

See below for a description of the method used to generate these results, and some older plots obtained with Adobe Camera Raw (ACR) as well as DPP.

Method
In making these measurements, it is firstly important to note that I used images in raw format from all cameras, and for the EOS 7D & 50D I disabled or checked that all the settings which might affect the basic output from the sensor were off (CF II functions 2, 3 & 4):

  • high ISO speed noise reduction
  • highlight tone priority
  • auto lighting optimiser

On the other cameras, I again made sure any similar options were off, although there are fewer available on the older models. Having done this, all the raw images should be as near as possible to what is coming off the sensor, with minimal processing. In other words a like for like comparison between the cameras, or 'level playing field'.

For these measurements I prepared and printed out a grey scale test chart - see below:

Grey scale test chart

Grey scale test chart for making measurements of noise level (click to download full resolution version in tiff format)

The purpose of this chart was to allow measurements of noise to be made for different image brightnesses from black through to white, to understand how the noise varied. The test chart was designed to have steps with different brightnesses, but each step was an even grey so that any variations in image brightness (or grey level) within the step would be due to noise and not the target.

To reduce any variations in image brightness due to printing imperfections in the test chart, I made sure all the images were slightly out of focus.

To process the raw files from each camera, I initially used Adobe Camera Raw, as downloaded for running in PhotoShop Elements versions 5 or 6. For the EOS 50D images, I used ACR 5.4, which is the latest version at the time of writing, together with PSE 6. For the EOS 350D and 40D images, which were done at the start of the 2009, I used an earlier version of ACR (4.5) with PSE 5. A quick check has shown no significant differences going back to the older EOS 350D and 40D shots and converting with ACR 5.4 in PSE 6.

In doing the conversion to 8-bit tiff files, I made sure any processing parameters that might increase or decrease noise were off (e.g. sharpening, noise reduction). I then used some freely downloadable ISee! software from BAM which is intended for the quantitative professional analysis of X-ray images. Other free packages may also be available to do this analysis.

This software allows the user to interactively define a small analysis area, that I set at a constant size of 30 x 30 pixels in all images. The software gives a readout of the standard deviation of the image grey levels within this area, which is a good measure of the noise level. I then moved this area across each picture, and measured the noise levels for each step, taking an average of five readings in slightly different positions on each step. The averaging was done to improve the accuracy of the measurements.

The example below shows the analysis of an image using ISee! in progress. The procedure for the analysis is as follows:

  1. Open required slightly out of focus image of the test chart above (having first printed it on a piece of photo-quality paper)
  2. Click the St button - highlighted in green below
  3. Draw out an analysis area using the mouse (left click top left, then drag out)
  4. Adjust the size of the area in the Statistics in window dialog box to be 30 x 30 - highlighted in blue below
  5. Pick up and move the area to the desired step on the image.
  6. Note down the mean and stdev values (circled in red below).
  7. Repeat stages 5-6 for about 5 different points on the current step.
  8. Repeat stages 5-7 for all steps.

Using the Isee! software from BAM for the analysis of the noise levels on an image of the step wedge.

Using the above procedure on different images, it is possible to build up a series of measurements that I could compare to study the variations in noise with various parameters, the main ones being image brightness or grey level, ISO setting, and of course camera type.

I should add that these measurements were made on a black & white versions of the tiff images (which is how they appear in the X-ray image analysis software), so that they presumably are measures of the luminance noise, and not chroma noise.

Raw converters
For this investigation, I initially used the raw converter available within PhotoShop Elements, which is called Adobe Camera Raw (ACR). However, part way through I also started looking at the results obtained with Canon's free Digital Photo Professional (DPP) that comes with Canon cameras. To my surprise there were significant differences between these two widely used raw converters. For more information on this, read on!

Measured noise levels
EOS 40D with ACR
The graph below shows how the noise levels for the EOS40D using ACR. The most obvious trend is for the noise at a given ISO setting to decrease with increasing image brightness (i.e. the lines slope down to the right). These lower noise levels in the brighter parts of the image may well be familiar to close observers of DSLR images. This agrees with my general qualitative experience that the noise is usually much more evident in the darker areas, than the lighter areas.

Also the graph shows that, as expected, increasing the ISO setting increases the overall noise levels, as expected. However there is a more subtle effect in that the variation in noise level with image brightness also changes with ISO setting and is much higher for ISO 1600 than ISO 400 (i.e. the noise in the darker areas of an ISO1600 shot is much worse than in the brighter areas, whereas at ISO400 the noise varies much less with image brightness).

The results below certainly lend support to the concept of keeping the exposure as high as possible to avoid under-exposed darker areas. This is apparently called ETTR (Expose To The Right), i.e. make sure your histogram of image values is over to the right (bright), not left (dark).

With raw files, I've also found that if the image is slightly clipped or saturated (spike at the right edge), then reducing the exposure in the raw converter gets over this problem, and the spike goes away. Very useful if you are trying to keep exposures up, and sometimes go too far!

With the 50D and 7D, there is about a full stop of leeway on the exposure adjustment. That is you can reduce the exposure in the raw converter by about a stop to avoid saturation. Beyond that, saturated areas just stay saturated. On the older 350D, there is somewhat less leeway.

Noise level measurements for the EOS 40D as a function of image brightness or grey level for three different ISO setttings (400, 800 and 1600)

EOS 350D, 40D and 50D noise levels compared (ACR)
The slightly confusing graph below shows results for all three cameras, at three different ISO settings. This shows the following:

  • For a given ISO setting, the EOS 40D had the lowest per pixel noise level, by about half a stop from the EOS 350D
  • The EOS 50D had by far the highest noise level, by about 1 stop compared with the 350D and 1.5 stops compared with the 40D.

The result for the 40D appears impressive, as it has pixels about 20% smaller in area than the 350D (10 Mpixels compared with 8Mpixels for the 350D). However, it is not quite as good as that, since the 40D is calibrated so that for a given ISO setting it is about 1/4 stop less sensitive than the 350D, so the overall improvement in noise is probably only about 1/4 stop.

The result for the 50D is poorer than I was expecting, and something of a surprise. Given it has almost twice as many pixels (15 Mpixels) in the same area of sensor as the 350D, the noise would be expected to increase. In fact given this increase in number of pixels by almost a factor of two, a one stop increase in noise levels is almost exactly what would be expected if all the noise were coming from photon counting statistics. But that is relative to the 350D. If you compare the 50D with the 40D, the comparison is less favourable. What happened to the claimed benefits of larger micro-lenses, increasing the effective area of each pixel on the 50D sensor?

Because of this suprising result, I looked to see what other measurements/comparisons of noise levels were available on the web. Perhaps there were errors in my results, or even my camera was a poor model. However, frustratingly there are very few direct comparisons to be found, and nearly always the exact details of the in-camera noise settings or post processing noise reduction settings aren't given, so it is impossible to know what is being compared with what. One exception to this is the excellent DP review website, which gives a true like for like comparison of raw images from the 40D and 50D, with no noise reduction in the raw conversion either. Sure enought the 50D images are noticeably noisier, but no attempt is made to measure the differences.

The site also make the following erudite comments:

"Removing any in-camera noise reduction and processing the images using Adobe Camera Raw (without NR) gives us the nearest thing to a 'level playing field' for assessing the relative noise levels of the two cameras' sensors. Despite the fact that the 50D is the newer camera it shows visibly more chroma and luminance noise than the 40D. Considering the 50D's much more tightly packed sensor (4.5 MP/cm² vs 3.1 MP/cm² on the 40D) this comes hardly as a surprise. It would have been unreasonable to expect Canon's engineers to overcome the laws of physics."

I'd agree with that, but it would have been useful to know by what factor the noise was higher on the 40D than the 50D, to see how it compares with the results below. Visual comparisons of their shots are difficult, but to my eyes the differences could be as high as 1 to 1.5 stops.

Noise level measurements for all three cameras as a function of image brightness or grey level for three different ISO setttings (400, 800 and 1600)

Adobe Camera Raw (ACR) compared with Digital Photo Professional (DPP) - on EOS50D
The results shown above for my newly acquired 50D came as quite a shock, and I wonder how useable the camera will be at anything above ISO 400. Even ISO 400 appears as noisy as ISO 800 on my 350D, which certainly caused me problems in the past on some shots. ISO 1600 was pretty much a setting of last resort for the 350D, which would equate to ISO 800 on the 50D. Not a good prospect!

Out of interest, I thought I would try as an experiment using the raw converter in Canon's free Digital Photo Professional (DPP V3.6.1) instead of my usual Adobe Camera Raw (ACR) in PhotoShop Elements. Imagine my surprise when I found that DPP was giving lower noise levels in the darker areas of the images than ACR, even with all the noise reduction turned off. The graph below shows the results for the 50D at ISO 400 and ISO 800. The improvement obtained using DPP is quite marked. For example at ISO 800, the noise in the darkest parts of the image with DPP is less than the noise at ISO 400 with ACR!

I'm unclear of the reasons for this difference. It could of course be that some hidden noise reduction is going on in DPP, without the user knowing about it. Whatever the reason, it appears better to use DPP than ACR for 50D images, certainly at the higher ISO settings.

Noise level measurements for the 50D as a function of image brightness or grey level - comparison between ACR and DPP raw converters

To show this makes a difference in practice, have a look at the example below, which shows a 1:1 crop of a somewhat underexposed pic of a Yellow-legged Gull, taken with the 50D at ISO 400. The chroma noise from ACR 5.4 (top) is noticeably worse than with DPP 3.6.1 (bottom) - just look at the darker areas on the back and wing tips. Even after using NeatImage (result not shown), the DPP result is appreciably better - NeatImage in my experience can struggle with chroma noise.

Results from ACR (top) and DPP (bottom) compared on a 1:1 crop of a distant Yellow-legged Gull [EOS 50D, ISO 400].
Note the higher chroma noise on the back with ACR than with DPP.

Here is another example, again 1:1 crops, with exposure +0.85 in both ACR and DPP. This was a distant bird, taken against the light, without the sun out, so the underparts were quite dark. Again to my eyes, the chroma noise is appreciably better in DPP (3.6.1) than ACR (5.4).

Hobby with ACR Hobby with DPP

Of course, I don't have a side-by side comparison obtained with the 350D (or my now ex 40D), but in my experience this sort of 1:1 noise level with the 50D is more characteristic of that seen at ISO 800 with the 350D, which is in accord with the measurements presented above.

Adobe Camera Raw (ACR) compared with Digital Photo Professional (DPP) - on EOS 350D
Given the apparent benefits found above for DPP compared with ACR using the EOS 50D, it seemed interesting to see what happens with the older EOS 350D. I presumed that there would be similar results. But it seems in this field that nothing is simple, or predictable!

The graph below shows the DPP and ACR noise levels at ISO 400, 800 and 1600 for the 350D. As for the 50D, the DPP results show lower noise levels in the darker areas of the image (to the left), but look what happens on the right hand side. Unlike for the 50D, DPP for the 350D gives noticeably higher noise levels than ACR for the brighter areas of the image - almost 1 stop worse in the brightest areas. In fact, over most image grey levels, ACR gives significantly lower noise levels than DPP!

Hence for the 350D, it seems that ACR is preferable to DPP - the reverse of the situation for the 50D. Again I have no idea for the explanation for these differences!

Noise level measurements for the EOS 350D as a function of image brightness or grey level - comparison between ACR and DPP raw converters

EOS 7D compared with EOS 50D and EOS 350D using Digital Photo Professional (DPP) New March 2011
With my recent (March 2011) acquisition of an EOS 7D, I was interested to see how its noise levels compared with the others. So see below for the noise level plot using Canon's DPP to convert the RAW files, which shows results from the 350D, 50D and 7D.

From the plot below, you can see that the noise levels on the 7D are quite impressive. Despite having slightly smaller pixels than the 50D, the 7D is giving significantly lower noise, especially at ISO 1600. The difference is smaller at the lower ISO settings, but is still noticeable in the darker areas of the image. The prehistoric 350D is similar to the much newer 50D, but of course only has about half the number of pixels of the 50D. Hence the 7D may be performing at a similar noise level to the much older 40D (which of course only had 10 Mpixels)!

These measurements do tend to confirm other reports - that Canon have improved the noise position on the EOS 7D, compared with the 50D. This is some reassurance that the expensive upgrade to the 7D may have been worthwhile! Time will tell if I continue to believe that!

However, what really matters to me is the noise left after the noise reduction software I use on all images - NeatImage. Of course that is a whole new subject in its own right! All I will say on that subject at present is that I often found that quite noisy ISO 800 50D images cleaned up really well with NeatImage, provided they were well exposed (i.e. ETTR).

It is very early days yet, but I'm not convinced NeatImage is doing quite such a good job on the 7D images at ISO 800. These appear to have quite a number of "rogue" pixels, either singly or in small groups, which are largely unaffected by NeatImage. So could it be that after NeatImage, the 50D is "better" than the 7D, despite its higher raw noise levels? Surely not?!

But at ISO 1600, the 7D images appear to clean up quite well with NeatImage, and overall I'm pretty impressed with the results at this high ISO setting.

Noise level measurements for the 350D, 50D and 7D as a function of image brightness or grey level for three different ISO settings (400, 800 and 1600)

Conclusions
So what if anything has all this shown, I can hear people saying already? Well its been a very interesting exercise for me, and I think the lessons learned from these results can be summarised as follows:

  • Use raw images all the time for best results and greater flexibility.
  • Especially at the higher ISO settings, with Adobe Camera Raw (ACR) the noise in the darker areas of a picture will be much higher than in the brighter areas. So the message here is keep the exposure up (ETTR).
  • If you clip it a bit (spike at right in histogram) don't worry too much. Just reduce the exposure in your raw converter and it will go away. This can be overdone, though, and very saturated images stay saturated. With the 50D and 7D, there is about a full stop of leeway on this adjustment, but beyond that saturated areas stay saturated no matter what exposure compensation is applied.
  • With ACR, the noise levels on the EOS 40D were the lowest measured - but effectively only about 1/4 stop better than those on the 350D, after allowing for differences in the calibrations used for setting the sensitivity of the ISO settings.
  • With ACR, the EOS 50D had by far the highest noise levels measured - about 1 stop worse than the 350D and a dismal 1.5 stops worse than the 40D.
  • Using the Canon Digital Photo Professional (DPP) instead of Adobe Camera Raw (ACR) for raw conversion gave better control of the noise levels in the darker areas of the 50D images, with no noise reduction set, and similar noise levels in the lighter areas.
  • Strangely, for the older EOS 350D camera, ACR was preferable to DPP, for unknown reasons.
  • With the new EOS 7D and DPP, the noise levels were measured to be significantly lower than the 50D and 350D especially at ISO 1600 - perhaps roughly back to the 40D levels!
  • There are surprisingly significant and unpredictable differences between the noise levels obtained with two very commonly used, and respected, raw converters (ACR, DPP). These appear to vary depending on the camera model.

Resolution
The other key image quality parameter of DSLR images is detail resolution - for some measurements with these cameras and two 400mm lenses, click here. Find out what benefit there is to the smaller sized pixels in the EOS50D!

Home | Trip Reports | Gallery | UK index | Oxon pics | UK pics | Dragonflies | Other Nature | Links

Fatbirder's Top 1000 Birding Websites