Testing

Introduction

This page collects resources and techniques for testing a CHDK build or port. See also Debugging and Testing Needed

this is a work in progress, please expand!

General guidelines

The CHDK User Manual gives an good overview of many features. Working through this list and verifying that your port matches the documented behavior.

Note that many CHDK features are poorly documented or poorly defined. When in doubt, consult the source.

Testing overrides

When testing overrides from script or the override menu, it is important to know Canon firmware displays and EXIF information may not reflect the actual value used. The functionality of the override should be determined from the image.

You should also verify that overrides are applied correctly when you press the shutter quickly without waiting for auto-focus.

Testing RAW/DNG

Verify DNGs validate with Adobe's DNG validator. dng_validate is included with the DNG SDK http://www.adobe.com/support/downloads/dng/dng_sdk.html

Take a series of shots and verify that each DNG contains the correct image data, to ensure raw buffers are all identified correctly.

Verify that a DNG is saved when you press the shutter quickly without waiting for auto-focus.

Check for shooting modes that do not produce valid raw data. Many Digic 4 and later do not produce valid raw in AUTO mode. Many cameras also have special half-resolution modes like ISO3200, Low Light or High ISO which do not produce valid raw. The CHDK port should show raw disabled in these modes.

File counter

Take a several raw or DNG and make sure that the file numbers match the jpeg, and that no raws are skipped. Try both continuous mode and single shots.

File folders

In the raw menu, set "RAW File in Dir with JPEG" and make sure the file is saved in the same directory as the corresponding jpeg.

Free RAM

Use Miscellaneous-> Show Memory info in the CHDK menu. If the Free Memory value is less than 400kb or so, there is a high risk of running out of memory. The port developer may be able to enable ARAM or EXMEM to gain additional memory.

Viewport buffers

To verify that the "viewport" is identified correctly, you can use the zebra, edge overlay and PTP live view functions

Zebra test

• Aim the camera at a bright light. The over exposed indicator should match the light, regardless of where the light appears on the screen.
• If it is offset, the viewport dimensions are probably wrong.
• If the zebra has no obvious relationship to the scene, the viewport address is probably wrong.

USB Remote

See USB Remote for details of this feature. A basic functionality test can be done using the standard USB cable and a PC:

• Start the camera without USB connected
• In CHDK Settings->Remote parameters set
  • Enable remote
  • Switch Type: One push
  • Control mode: Normal
• Switch the camera to shooting mode, and select a still capture mode such as P
• Plug in the USB cable. The camera should act as if the shutter was pressed half way
• Unplug the USB cable. The camera should shoot.
• Enable remote sync
• Plug in the USB cable. The screen should go dark, and a status LED may blink.
• Unplug the USB cable. The camera should shoot.

Don't forget to disable the remote if you want to use regular USB file transfer or CHDK PTP.

Testing with scripts

There are some scripts which exercise various CHDK capabilities.

The scripts listed below are included with a "full" CHDK install, usually in the CHDK/SCRIPTS/TEST directory.

isobase.lua

This checks whether the base ISO used for CHDK overrides need to be defined for the port. Run the script. If it reports "set CAM_MARKET_ISO_BASE ..." needed, make sure CAM_MARKET_ISO_BASE is present with the correct value in the ports platform_camera.h. If it reports unknown value, there may be an error in the test, or the define may need to be added with a new value. If It reports 100, no define is needed.

Note: This script only checks whether the #define is required, it doesn't tell you it is already defined correctly or not.

mftest.lua

This script tests focus (subject distance) override capabilities for CHDK 1.3 and later. This script should be used on every new port to determine the which override methods work. The script is included in full builds as CHDK/SCRIPTS/TEST/MFTEST.LUA

Test procedure for new ports

• Set the camera to P mode
• Make sure Canon firmware MF and AF lock are not enabled
• Make sure AF and Servo AF are disabled in the Canon firmware shooting menu, if present.
• Enable all test methods and set "Bypass Interlocks?" to Yes in the script options.
• Run the script.
• If the camera crashes on a test, disable that test and repeat

A detailed log is recored in mf_<cameraname>.csv (the name may be truncated on dryos cameras).

The log is appended on each run, so a full record will be present if you have to run it multiple times.

If a test is listed as passed, the method works.

If a test is listed as failed, the method either doesn't work, or the difference between the requested value and the value the camera set exceeded the tolerance in the script. It should be possible to distinguish this in the log.

Working methods should be enabled in the port's platform_camera.h using the CAM_SD_OVER_IN_* defines.

llibtst.lua

Tests the lua standard libraries. This also exercises many C library functions.

• Documentation: Lua#Lua_library_test

Test procedure

• run the script, with all of the 'skip' options set to zero.
• If all test succeeded, it will print "end ... OK" for each item.
• If any tests failed, examine A/llibtst.log

setmode.lua

Tests the capture mode switching functions and mode map. This script is included with a "full" CHDK install.

• Documentation: Lua/Scripts:Standard/Test/Setmode

Test procedure

• start the camera in play mode
• run the script with all options except "mode change delay" set to zero
• the camera should switch to record, and switch rapidly between each known shooting mode
• If everything was OK, it will print pass
• If something failed, examine A/setmode.log

ubtest.bas

General purpose ubasic test.

• Documentation: http://chdk.setepontos.com/index.php?topic=6592.0

Test procedure

• Start the camera in play mode
• Start the script with interactive = 1 and sleep between tests = 5
• The message "key test - set to end" will appear
• If you press up/down/left/right/menu/erase the corresponding key name should be printed.
• Press set to go to the next test
• The canon menu should open and selection change a couple times
• The camera will switch to record mode, and switch to P mode
• The camera will take shots as follows:
  • auto exposure, using get_shooting, press shoot_half, press shoot_full etc
  • auto exposure + or - 2 stops Tv, using shoot
  • auto exposure + or - 2 stops Sv, using shoot
  • If ND filter is present, one shot each with ND forced in and out, using shoot
• If no failures were identified, prints "ok", otherwise failed.
• Output is logged to CHDK/LOGS/LOG_0001.TXT. The log will indicate whether the shots above were + or -, with the corresponding exposure number. Compare the jpegs to ensure the images show the expected variation in exposure.

MD_tune.bas

Tests the reaction speed of motion detection. This can be used find the optimal implementation of vid_get_viewport_live_fb()

• Documentation http://chdk.setepontos.com/index.php?topic=9366.msg96633#msg96633 (note the scripts, patches and defines in the thread are no longer needed, you should use the script included with the CHDK source above)

Prerequisites

• Live view buffers functions return pointers to valid data (see other frame viewport buffers above)
• CAM_AF_LED is defined as camera_set_led identifier for AF in platform_camera.h
• camera_set_led is implemented and supports trigger AF

Test procedure

• Put the camera in record mode, aimed at a nearby subject in an environment dark enough for the AF led to show up well
• Run the script
• Check that MD is triggering reliably when the AF blinks and note the min/max/and average values at the top of the screen. These values may be obscured by the CHDK OSD, turn the OSD off to avoid this.
• If alternative implementations of vid_get_viewport_live_fb are available (i.e. using active_viewport_buffer as the current, next or previous frame), try each one and use the one that gives the best results.

Note - As of build 3986, statistics are cleared each time the script is run. Previously, a reboot was required.

hooktest.lua

This script tests the operation and placement of the remote hook (wait_until_remote_button_is_released) raw hook (capt_seq_hook_raw_here), and override code (shooting_expo_param_override).

It attempts to verify the file counter (get_exp_count etc) and check whether PAUSE_FOR_FILE_COUNTER is required, but this may not be definitive so the file counter checks described in the RAW/DNG seciton above should also be done.

• Documentation http://chdk.setepontos.com/index.php?topic=12473.msg123637#msg123637

Prerequisites

• capt_seq_task should be implemented with all the hooks.

Test procedure

• Enable continuous mode in the Canon UI
• Run the script
• If the output says FAILED, examine the log.

Interpreting results

If a hook isn't reached, the single shot test will generate errors like:

0020660: FAIL: hook_raw count 0 expect 1

and the fast and cont tests will generate errors like:

0037700: FAIL: hook_raw wait timeout

If the file counter isn't incremented when the raw hook is entered, you will get something like

0527810: FAIL: exp count 9812 expect 9813

NOTE Some older cameras (probably Digic II vxworks, maybe some digic III and early dryos) will fail the file counter checks in the 'cont' and 'fast' tests. See the forum thread for details.