Stephen Burch's Birding & Dragonfly Website

Home | Trip Reports | Gallery | UK index | Oxon pics | UK pics | Dragonflies | Other Nature | Links

Quantitative measurements of DLSR raw image noise levels

Introduction
For some time now, I've been interested in the noise that appears on DSLR images, and the best methods for getting rid of it (see my comments about NeatImage for example on another page on this website). However, on the principle that knowing as much as possible about one's enemy is a good idea, I thought I'd do a bit more investigation of the subject.

Simply looking at digital camera pictures and trying to make valid assessments and comparisons of noise levels may have its place, but the results can be very subjective and depend hugely on the subject matter of the picture, as well as the camera used and all the settings involved in getting the image out of the camera onto the screen. The size the image is viewed at can also makes a big difference. An image at 20% magnification can appear virtually noise free, but show it at 1:1 and up comes the noise.

Hence, I have taken a different approach and developed a reasonably simple method for quantitatively measuring pixel to pixel noise levels on digital camera pictures, effectively analysed at 1:1 magnification. I've then applied this method to images taken with different Canon DSLR cameras at different ISO settings, with some interesting results (to me anyway!).

The cameras assessed to date are the EOS 350D, EOS 40D and EOS 50D.

Method
In making these measurements, it is firstly important to note that I used images in raw format from all cameras, and for the EOS 50D I disabled or checked that all the settings which might affect the basic output from the sensor were off (CF II functions 2, 3 & 4):

• high ISO speed noise reduction
• highlight tone priority
• auto lighting optimiser

On the other cameras, I again made sure any similar options were off, although there are fewer available on the older models. Having done this, all the raw images should be as near as possible to what is coming off the sensor, with minimal processing. In other words a like for like comparison between the cameras, or 'level playing field'.

For these measurements I prepared and printed out a grey scale test chart - see below:

The purpose of this chart was to allow measurements of noise to be made for different image brightnesses from black through to white, to understand how the noise varied. The test chart was designed to have steps with different brightnesses, but each step was an even grey so that any variations in image brightness (or grey level) within the step would be due to noise and not the target.

To reduce any variations in image brightness due to printing imperfections in the test chart, I made sure all the images were slightly out of focus.

To process the raw files from each camera, I initially used Adobe Camera Raw, as downloaded for running in PhotoShop Elements versions 5 or 6. For the EOS 50D images, I used ACR 5.4, which is the latest version at the time of writing, together with PSE 6. For the EOS 350D and 40D images, which were done at the start of the 2009, I used an earlier version of ACR (4.5) with PSE 5. A quick check has shown no significant differences going back to the older EOS 350D and 40D shots and converting with ACR 5.4 in PSE 6.

In doing the conversion to 8-bit tiff files, I made sure any processing parameters that might increase or decrease noise were off (e.g. sharpening, noise reduction). I then used some freely downloadable software which is intended for the quantitative professional analysis of X-ray images. This allows the user to interactively define a small analysis area, that I set at a constant size of 30 x 30 pixels in all images. The software gives a readout of the standard deviation of the image grey levels within this area, which is a good measure of the noise level. I then moved this area across each picture, and measured the noise levels for each step, taking an average of five readings in slightly different positions on each step. The averaging was done to improve the accuracy of the measurements.

This gave me a series of measurements I could compare to study the variations in noise with various parameters, the main ones being image brightness or grey level, ISO setting, and of course camera type.

I should add that these measurements were made on a black & white versions of the tiff images (which is how they appear in the X-ray image analysis software), so that they presumably are measures of the luminance noise, and not chroma noise.

Measured noise levels
EOS 40D
The graph below shows how the noise levels for the EOS40D for a given ISO setting tend to decrease with increasing image brightness (i.e. the lines slope down to the right). These lower noise levels in the brighter parts of the image may well be familiar to close observers of DSLR images (or at least those obtained with Adobe Camera Raw - see below for more on that).

Certainly in my experience the noise is often much higher in the darker areas - as shown below. The variation in noise level with image brightness also changes with ISO setting and is higher for ISO 1600 than ISO 400 (i.e. the noise in the darker areas of an ISO1600 shot is much worse than in the brighter areas). Also of course increasing the ISO setting increases the overall noise levels, as expected.

The results below certainly lend support to the concept of keeping the exposure as high as possible to avoid under-exposed darker areas. This is apparently called ETTR (Expose To The Right), i.e. make sure your histogram of image values is over to the right (bright), not left (dark).

With raw files, I've also found that if the image is slightly clipped or saturated (spike at the right edge), then reducing the exposure in the raw converter gets over this problem, and the spike goes away. Very useful if you are trying to keep exposures up, and sometimes go too far!

EOS 350D, 40D and 50D compared
The slightly confusing graph below shows results for all three cameras, at three different ISO settings. This shows the following:

• For a given ISO setting, the EOS 40D had the lowest per pixel noise level, by about half a stop from the EOS 350D
• The EOS 50D had by far the highest noise level, by about 1 stop compared with the 350D and 1.5 stops compared with the 40D.

The result for the 40D appears impressive, as it has pixels about 20% smaller in area than the 350D (10 Mpixels compared with 8Mpixels for the 350D). However, it is not quite as good as that, since the 40D is calibrated so that for a given ISO setting it is about 1/4 stop less sensitive than the 350D, so the overall improvement in noise is probably only about 1/4 stop.

The result for the 50D is poorer than I was expecting, and something of a surprise. Given it has almost twice as many pixels (15 Mpixels) in the same area of sensor as the 350D, the noise would be expected to increase. In fact given this increase in number of pixels by almost a factor of two, a one stop increase in noise levels is almost exactly what would be expected if all the noise were coming from photon counting statistics. But that is relative to the 350D. If you compare the 50D with the 40D, the comparison is less favourable. What happened to the claimed benefits of larger micro-lenses, increasing the effective area of each pixel on the 50D sensor?

Because of this suprising result, I looked to see what other measurements/comparisons of noise levels were available on the web. Perhaps there were errors in my results, or even my camera was a poor model. However, frustratingly there are very few direct comparisons to be found, and nearly always the exact details of the in-camera noise settings or post processing noise reduction settings aren't given, so it is impossible to know what is being compared with what. One exception to this is the excellent DP review website, which gives a true like for like comparison of raw images from the 40D and 50D, with no noise reduction in the raw conversion either. Sure enought the 50D images are noticeably noisier, but no attempt is made to measure the differences.

The site also make the following erudite comments:

"Removing any in-camera noise reduction and processing the images using Adobe Camera Raw (without NR) gives us the nearest thing to a 'level playing field' for assessing the relative noise levels of the two cameras' sensors. Despite the fact that the 50D is the newer camera it shows visibly more chroma and luminance noise than the 40D. Considering the 50D's much more tightly packed sensor (4.5 MP/cm² vs 3.1 MP/cm² on the 40D) this comes hardly as a surprise. It would have been unreasonable to expect Canon's engineers to overcome the laws of physics."

I'd agree with that, but it would have been useful to know by what factor the noise was higher on the 40D than the 50D, to see how it compares with the results below. Visual comparisons of their shots are difficult, but to my eyes the differences could be as high as 1 to 1.5 stops.

Adobe Camera Raw (ACR) compared with Digital Photo Professional (DPP) - on EOS50D
The results shown above for my newly acquired 50D came as quite a shock, and I wonder how useable the camera will be at anything above ISO 400. Even ISO 400 appears as noisy as ISO 800 on my 350D, which certainly caused me problems in the past on some shots. ISO 1600 was pretty much a setting of last resort for the 350D, which would equate to ISO 800 on the 50D. Not a good prospect!

Out of interest, I thought I would try as an experiment using the raw converter in Canon's free Digital Photo Professional (DPP V3.6.1) instead of my usual Adobe Camera Raw (ACR) in PhotoShop Elements. Imagine my surprise when I found that DPP was giving lower noise levels in the darker areas of the images than ACR, even with all the noise reduction turned off. The graph below shows the results for the 50D at ISO 400 and ISO 800. The improvement obtained using DPP is quite marked. For example at ISO 800, the noise in the darkest parts of the image with DPP is less than the noise at ISO 400 with ACR!

I'm unclear of the reasons for this difference. It could of course be that some hidden noise reduction is going on in DPP, without the user knowing about it. Whatever the reason, it appears better to use DPP than ACR for 50D images, certainly at the higher ISO settings.

To show this makes a difference in practice, have a look at the example below, which shows a 1:1 crop of a somewhat underexposed pic of a Yellow-legged Gull, taken with the 50D at ISO 400. The chroma noise from ACR 5.4 (top) is noticeably worse than with DPP 3.6.1 (bottom) - just look at the darker areas on the back and wing tips. Even after using NeatImage (result not shown), the DPP result is appreciably better - NeatImage in my experience can struggle with chroma noise.

Here is another example, again 1:1 crops, with exposure +0.85 in both ACR and DPP. This was a distant bird, taken against the light, without the sun out, so the underparts were quite dark. Again to my eyes, the chroma noise is appreciably better in DPP (3.6.1) than ACR (5.4).

Hobby with ACR	Hobby with DPP

Of course, I don't have a side-by side comparison obtained with the 350D (or my now ex 40D), but in my experience this sort of 1:1 noise level with the 50D is more characteristic of that seen at ISO 800 with the 350D, which is in accord with the measurements presented above.

Adobe Camera Raw (ACR) compared with Digital Photo Professional (DPP) - on EOS 350D
Given the apparent benefits found above for DPP compared with ACR using the EOS 50D, it seemed interesting to see what happens with the older EOS 350D. I presumed that there would be similar results. But it seems in this field that nothing is simple, or predictable!

The graph below shows the DPP and ACR noise levels at ISO 400, 800 and 1600 for the 350D. As for the 50D, the DPP results show lower noise levels in the darker areas of the image (to the left), but look what happens on the right hand side. Unlike for the 50D, DPP for the 350D gives noticeably higher noise levels than ACR for the brighter areas of the image - almost 1 stop worse in the brightest areas. In fact, over most image grey levels, ACR gives significantly lower noise levels than DPP!

Hence for the 350D, it seems that ACR is preferable to DPP - the reverse of the situation for the 50D. Again I have no idea for the explanation for these differences!

Conclusions
So what if anything has all this shown, I can hear people saying already? Well its been a very interesting exercise for me, and I think the lessons learned from these results can be summarised as follows:

• Use raw images all the time for best results and greater flexibility.
• Especially at the higher ISO settings, with Adobe Camera Raw (ACR) the noise in the darker areas of a picture will be much higher than in the brighter areas. So the message here is keep the exposure up (ETTR).
• If you clip it a bit (spike at right in histogram) don't worry too much. Just reduce the exposure in your raw converter and it will go away. This can be overdone, though, and very saturated images stay saturated.
• With ACR, the noise levels on the EOS 40D were the lowest measured - but effectively only about 1/4 stop better than those on the 350D, after allowing for differences in the calibrations used for setting the sensitivity of the ISO settings.
• With ACR, the EOS 50D had by far the highest noise levels measured - about 1 stop worse than the 350D and a dismal 1.5 stops worse than the 40D.
• Using the Canon Digital Photo Professional (DPP) instead of Adobe Camera Raw (ACR) for raw conversion gave better control of the noise levels in the darker areas of the 50D images, with no noise reduction set, and similar noise levels in the lighter areas.
• Strangely, for the older EOS 350D camera, ACR was preferable to DPP, for unknown reasons.
• There are surprisingly significant and unpredictable differences between the noise levels obtained with two very commonly used, and respected, raw converters (ACR, DPP). These appear to vary depending on the camera model.

I'd be interested in any comments from others on this. Do other more experienced users of 50D raw images suffer from high noise levels?

Resolution
The other key image quality parameter of DSLR images is detail resolution - for some measurements with these cameras and two 400mm lenses, click here. Find out what benefit there is to the smaller sized pixels in the EOS50D!

Home | Trip Reports | Gallery | UK index | Oxon pics | UK pics | Dragonflies | Other Nature | Links