1240116SmarcelMajor changes between releases Automated Testing Framework 2240116Smarcel=========================================================================== 3240116Smarcel 4273929SjmmvChanges in version 0.21 5273929Sjmmv*********************** 6240116Smarcel 7273929SjmmvReleased on October 23rd, 2014. 8273929Sjmmv 9273929Sjmmv* Restored the atf(7) manual page to serve as a reference to all the other 10273929Sjmmv manual pages shipped by ATF. 11273929Sjmmv 12273929Sjmmv* Added the -s flag to atf-sh to support specifying the shell interpreter 13273929Sjmmv to be used. 14273929Sjmmv 15273929Sjmmv* Removed ATF_WORKDIR. The only remaining consumers have been converted to 16273929Sjmmv use the standard TMPDIR environment variable. As a benefit, and because 17273929Sjmmv Kyua forces the TMPDIR to live within the test case's work directory, 18273929Sjmmv any stale files left behind by ATF will be automatically cleaned up. 19273929Sjmmv 20273929Sjmmv* Documented the environment variables recognized by each component in the 21273929Sjmmv relevant manual pages. This information was lost with the atf-config(1) 22273929Sjmmv removal. 23273929Sjmmv 24273929Sjmmv* Added a new "require.diskspace" metadata property to test cases so that 25273929Sjmmv they can specify the minimum amount of disk space required for the test 26273929Sjmmv to run. 27273929Sjmmv 28273929Sjmmv* Renamed the atf-{c,c++,sh}-api(3) manual pages to atf-{c,c++,sh}(3) for 29273929Sjmmv discoverability purposes. Symbolic links are provided for the time 30273929Sjmmv being to still make the old names visible. 31273929Sjmmv 32273929Sjmmv* Issue #5: Recommend the (expected, actual) idiom for calls to the test 33273929Sjmmv macros in the manual pages. 34273929Sjmmv 35273929Sjmmv* Issue #7: Stopped catching unhandled exceptions in atf-c++ tests. This 36273929Sjmmv propagates the crash to the caller, which in turn allows it to obtain 37273929Sjmmv proper debugging information. In particular, Kyua should now be able to 38273929Sjmmv extract a stacktrace pinpointing the problem. 39273929Sjmmv 40273929Sjmmv* Issue #8: Fixed atf-c/macros_test:use test failures spotted by the clang 41273929Sjmmv that ships with FreeBSD 11.0-CURRENT. 42273929Sjmmv 43273929Sjmmv* Issue #12: Improved documentation of atf-sh(3) and atf-check(1) by better 44273929Sjmmv explaining how they relate to each other. 45273929Sjmmv 46273929Sjmmv* Issue #14: Stopped setting 'set -e' in atf-sh. This setting was 47273929Sjmmv initially added as a way to enable a "strict" mode in the library and to 48273929Sjmmv make test cases fail fast when they run unprotected commands. However, 49273929Sjmmv doing so in the library is surprising as the responsibility of enabling 50273929Sjmmv 'set -e' should be on the user's code. Also, 'set -e' introduces 51273929Sjmmv inconsistent behavior on subshells and users do not expect that. 52273929Sjmmv 53273929Sjmmv* Issue #15: Fixed atf_utils_{fork,wait} to support nested calls. 54273929Sjmmv 55273929Sjmmv* Issue #16: Fixed test failures (by removing a long-standing hack) on 56273929Sjmmv systems that lack \e support in printf(1). 57273929Sjmmv 58273929Sjmmv* Issue #19: Removed stale references to atf-config and atf-run. 59273929Sjmmv 60273929Sjmmv 61261897SjmmvChanges in version 0.20 62261897Sjmmv*********************** 63261897Sjmmv 64261897SjmmvExperimental version released on February 7th, 2014. 65261897Sjmmv 66261897SjmmvThis is the first release without the code for the deprecated tools. If 67261897Sjmmvyou require such code, please fetch a copy of the 0.19 release and extract 68261897Sjmmvthe 'tools' directory for your own consumption. 69261897Sjmmv 70261897Sjmmv* Removed the deprecated tools. This includes atf-config, atf-report, 71261897Sjmmv atf-run and atf-version. 72261897Sjmmv 73271875Srodrigc* Issue #8: Fixed atf-c/macros_test:use test failures spotted by the clang 74271875Srodrigc that ships with FreeBSD 11.0-CURRENT. 75261897Sjmmv 76271875Srodrigc 77261897SjmmvChanges in version 0.19 78261897Sjmmv*********************** 79261897Sjmmv 80261897SjmmvExperimental version released on February 7th, 2014. 81261897Sjmmv 82261897SjmmvThis is the last release to bundle the code for the deprecated tools. 83261897SjmmvThe next release will drop their code and will stop worrying about 84261897Sjmmvbackwards compatibility between the ATF libraries and what the old tools 85261897Sjmmvmay or may not support. 86261897Sjmmv 87261897SjmmvIf you still require the old tools for some reason, grab a copy of the 88261897Sjmmv'tools' directory now. The code in this directory is standalone and 89261897Sjmmvdoes not depend on any internal details of atf-c++ any longer. 90261897Sjmmv 91261897Sjmmv* Various fixes and improvements to support running as part of the FreeBSD 92261897Sjmmv test suite. 93261897Sjmmv 94261897Sjmmv* Project hosting moved from Google Code (as a subproject of Kyua) to 95261897Sjmmv GitHub (as a first-class project). The main reason for the change is 96261897Sjmmv the suppression of binary downloads in Google Code on Jan 15th, 2014. 97261897Sjmmv See https://github.com/jmmv/atf/ 98261897Sjmmv 99261897Sjmmv* Removed builtin help from atf-sh(1) and atf-check(1) for simplicity 100261897Sjmmv reasons. In other words, their -h option is gone. 101261897Sjmmv 102261897Sjmmv* Moved the code of the deprecated tools into a 'tools' directory and 103261897Sjmmv completely decoupled their code from the internals of atf-c++. The 104261897Sjmmv reason for this is to painlessly allow a third-party to maintain a 105261897Sjmmv copy of these tools after we delete them because upcoming changes to 106261897Sjmmv atf-c++ would break the stale tools. 107261897Sjmmv 108261897Sjmmv 109258289SjmmvChanges in version 0.18 110258289Sjmmv*********************** 111258289Sjmmv 112258289SjmmvExperimental version released on November 16th, 2013. 113258289Sjmmv 114258289Sjmmv* Issue 45: Added require.memory support in atf-run for FreeBSD. 115258289Sjmmv 116258289Sjmmv* Fixed an issue with the handling of cin with libc++. 117258289Sjmmv 118258289Sjmmv* Issue 64: Fixed various mandoc formatting warnings. 119258289Sjmmv 120258289Sjmmv* NetBSD PR bin/48284: Made atf-check flush its progress message to 121258289Sjmmv stdout so that an interrupted test case always shows the last message 122258289Sjmmv being executed. 123258289Sjmmv 124258289Sjmmv* NetBSD PR bin/48285: Fixed atf_check examples in atf-sh-api(3). 125258289Sjmmv 126258289Sjmmv 127258289SjmmvChanges in version 0.17 128258289Sjmmv*********************** 129258289Sjmmv 130258289SjmmvExperimental version released on February 14th, 2013. 131258289Sjmmv 132258289Sjmmv* Added the atf_utils_cat_file, atf_utils_compare_file, 133258289Sjmmv atf_utils_copy_file, atf_utils_create_file, atf_utils_file_exists, 134258289Sjmmv atf_utils_fork, atf_utils_grep_file, atf_utils_grep_string, 135258289Sjmmv atf_utils_readline, atf_utils_redirect and atf_utils_wait utility 136258289Sjmmv functions to atf-c-api. Documented the already-public 137258289Sjmmv atf_utils_free_charpp function. 138258289Sjmmv 139258289Sjmmv* Added the cat_file, compare_file, copy_file, create_file, file_exists, 140258289Sjmmv fork, grep_collection, grep_file, grep_string, redirect and wait 141258289Sjmmv functions to the atf::utils namespace of atf-c++-api. These are 142258289Sjmmv wrappers around the same functions added to the atf-c-api library. 143258289Sjmmv 144258289Sjmmv* Added the ATF_CHECK_MATCH, ATF_CHECK_MATCH_MSG, ATF_REQUIRE_MATCH and 145258289Sjmmv ATF_REQUIRE_MATCH_MSG macros to atf-c to simplify the validation of a 146258289Sjmmv string against a regular expression. 147258289Sjmmv 148258289Sjmmv* Miscellaneous fixes for manpage typos and compilation problems with 149258289Sjmmv clang. 150258289Sjmmv 151258289Sjmmv* Added caching of the results of those configure tests that rely on 152258289Sjmmv executing a test program. This should help crossbuild systems by 153258289Sjmmv providing a mechanism to pre-specify what the results should be. 154258289Sjmmv 155258289Sjmmv* PR bin/45690: Make atf-report convert any non-printable characters to 156258289Sjmmv a plain-text representation (matching their corresponding hexadecimal 157258289Sjmmv entities) in XML output files. This is to prevent the output of test 158258289Sjmmv cases from breaking xsltproc later. 159258289Sjmmv 160258289Sjmmv 161240116SmarcelChanges in version 0.16 162240116Smarcel*********************** 163240116Smarcel 164240116SmarcelExperimental version released on July 10th, 2012. 165240116Smarcel 166240116Smarcel* Added a --enable-tools flag to configure to request the build of the 167240116Smarcel deprecated ATF tools, whose build is now disabled by default. In order 168240116Smarcel to continue running tests, you should migrate to Kyua instead of enabling 169240116Smarcel the build of the deprecated tools. The kyua-atf-compat package provides 170240116Smarcel transitional compatibility versions of atf-run and atf-report built on 171240116Smarcel top of Kyua. 172240116Smarcel 173240116Smarcel* Tweaked the ATF_TEST_CASE macro of atf-c++ so that the compiler can 174240116Smarcel detect defined but unused test cases. 175240116Smarcel 176240116Smarcel* PR bin/45859: Fixed some XSLT bugs that resulted in the tc-time and 177240116Smarcel tp-time XML tags leaking into the generated HTML file. Also improved 178240116Smarcel the CSS file slightly to correct alignment and color issues with the 179240116Smarcel timestamps column. 180240116Smarcel 181240116Smarcel* Optimized atf-c++/macros.hpp so that GNU G++ consumes less memory during 182240116Smarcel compilation with GNU G++. 183240116Smarcel 184240116Smarcel* Flipped the default to building shared libraries for atf-c and atf-c++, 185240116Smarcel and started versioning them. As a side-effect, this removes the 186240116Smarcel --enable-unstable-shared flag from configure that appears to not work any 187240116Smarcel more (under NetBSD). Additionally, some distributions require the use of 188240116Smarcel shared libraries for proper dependency tracking (e.g. Fedora), so it is 189240116Smarcel better if we do the right versioning upstream. 190240116Smarcel 191240116Smarcel* Project hosting moved from an adhoc solution (custom web site and 192240116Smarcel Monotone repository) to Google Code (standard wiki and Git). ATF now 193240116Smarcel lives in a subcomponent of the Kyua project. 194240116Smarcel 195240116Smarcel 196240116SmarcelChanges in version 0.15 197240116Smarcel*********************** 198240116Smarcel 199240116SmarcelExperimental version released on January 16th, 2012. 200240116Smarcel 201240116Smarcel* Respect stdin in atf-check. The previous release silenced stdin for any 202240116Smarcel processes spawned by atf, not only test programs, which caused breakage 203240116Smarcel in tests that pipe data through atf-check. 204240116Smarcel 205240116Smarcel* Performance improvements to atf-sh. 206240116Smarcel 207240116Smarcel* Enabled detection of unused parameters and variables in the code and 208240116Smarcel fixed all warnings. 209240116Smarcel 210240116Smarcel* Changed the behavior of "developer mode". Compiler warnings are now 211240116Smarcel enabled unconditionally regardless of whether we are in developer mode or 212240116Smarcel not; developer mode is now only used to perform strict warning checks and 213240116Smarcel to enable assertions. Additionally, developer mode is now only 214240116Smarcel automatically enabled when building from the repository, not for formal 215240116Smarcel releases. 216240116Smarcel 217240116Smarcel* Added new Autoconf M4 macros (ATF_ARG_WITH, ATF_CHECK_C and 218240116Smarcel ATF_CHECK_CXX) to provide a consistent way of defining a --with-arg flag 219240116Smarcel in configure scripts and detecting the presence of any of the ATF 220240116Smarcel bindings. Note that ATF_CHECK_SH was already introduced in 0.14, but it 221240116Smarcel has now been modified to also honor --with-atf if instantiated. 222240116Smarcel 223240116Smarcel* Added timing support to atf-run / atf-report. 224240116Smarcel 225240116Smarcel* Added support for a 'require.memory' property, to specify the minimum 226240116Smarcel amount of physical memory needed by the test case to yield valid results. 227240116Smarcel 228240116Smarcel* PR bin/45690: Force an ISO-8859-1 encoding in the XML files generated by 229240116Smarcel atf-report so that invalid data in the output of test cases does not 230240116Smarcel mangle our report. 231240116Smarcel 232240116Smarcel 233240116SmarcelChanges in version 0.14 234240116Smarcel*********************** 235240116Smarcel 236240116SmarcelExperimental version released on June 14th, 2011. 237240116Smarcel 238240116Smarcel* Added a pkg-config file for atf-sh and an aclocal file to ease the 239240116Smarcel detection of atf-sh from autoconf scripts. 240240116Smarcel 241240116Smarcel* Made the default test case body defined by atf_sh fail. This is to 242240116Smarcel ensure that test cases are properly defined in test programs and helps 243240116Smarcel in catching typos in the names of the body functions. 244240116Smarcel 245240116Smarcel* PR bin/44882: Made atf-run connect the stdin of test cases to /dev/zero. 246240116Smarcel This provides more consistent results with "normal" execution (in 247240116Smarcel particular, when tests are executed detached from a terminal). 248240116Smarcel 249240116Smarcel* Made atf-run hardcode TZ=UTC for test cases. It used to undefine TZ, but 250240116Smarcel that does not take into account that libc determines the current timezone 251240116Smarcel from a configuration file. 252240116Smarcel 253240116Smarcel* All test programs will now print a warning when they are not run through 254240116Smarcel atf-run(1) stating that this is unsupported and may deliver incorrect 255240116Smarcel results. 256240116Smarcel 257240116Smarcel* Added support for the 'require.files' test-case property. This allows 258240116Smarcel test cases to specify installed files that must be present for the test 259240116Smarcel case to run. 260240116Smarcel 261240116Smarcel 262240116SmarcelChanges in version 0.13 263240116Smarcel*********************** 264240116Smarcel 265240116SmarcelExperimental version released on March 31st, 2011. 266240116Smarcel 267240116SmarcelThis is the first release after the creation of the Kyua project, a more 268240116Smarcelmodular and reliable replacement for ATF. From now on, ATF will change to 269240116Smarcelaccomodate the transition to this new codebase, but ATF will still continue 270240116Smarcelto see development in the short/medium term. Check out the project page at 271240116Smarcelhttp://code.google.com/p/kyua/ for more details. 272240116Smarcel 273240116SmarcelThe changes in this release are: 274240116Smarcel 275240116Smarcel* Added support to run the tests with the Kyua runtime engine (kyua-cli), a 276240116Smarcel new package that aims to replace atf-run and atf-report. The ATF tests 277240116Smarcel can be run with the new system by issuing a 'make installcheck-kyua' from 278240116Smarcel the top-level directory of the project (assuming the 'kyua' binary is 279240116Smarcel available during the configuration stage of ATF). 280240116Smarcel 281240116Smarcel* atf-run and atf-report are now in maintenance mode (but *not* deprecated 282240116Smarcel yet!). Kyua already implements a new, much more reliable runtime engine 283240116Smarcel that provides similar features to these tools. That said, it is not 284240116Smarcel complete yet so all development efforts should go towards it. 285240116Smarcel 286240116Smarcel* If GDB is installed, atf-run dumps the stack trace of crashing test 287240116Smarcel programs in an attempt to aid debugging. Contributed by Antti Kantee. 288240116Smarcel 289240116Smarcel* Reverted default timeout change in previous release and reset its value 290240116Smarcel to 5 minutes. This was causing several issues, specially when running 291240116Smarcel the existing NetBSD test suite in qemu. 292240116Smarcel 293240116Smarcel* Fixed the 'match' output checker in atf-check to properly validate the 294240116Smarcel last line of a file even if it does not have a newline. 295240116Smarcel 296240116Smarcel* Added the ATF_REQUIRE_IN and ATF_REQUIRE_NOT_IN macros to atf-c++ to 297240116Smarcel check for the presence (or lack thereof) of an element in a collection. 298240116Smarcel 299240116Smarcel* PR bin/44176: Fixed a race condition in atf-run that would crash atf-run 300240116Smarcel when the cleanup of a test case triggered asynchronous modifications to 301240116Smarcel its work directory (e.g. killing a daemon process that cleans up a pid 302240116Smarcel file in the work directory). 303240116Smarcel 304240116Smarcel* PR bin/44301: Fixed the sample XSLT file to report bogus test programs 305240116Smarcel instead of just listing them as having 0 test cases. 306240116Smarcel 307240116Smarcel 308240116SmarcelChanges in version 0.12 309240116Smarcel*********************** 310240116Smarcel 311240116SmarcelExperimental version released on November 7th, 2010. 312240116Smarcel 313240116Smarcel* Added the ATF_REQUIRE_THROW_RE to atf-c++, which is the same as 314240116Smarcel ATF_REQUIRE_THROW but allows checking for the validity of the exception's 315240116Smarcel error message by means of a regular expression. 316240116Smarcel 317240116Smarcel* Added the ATF_REQUIRE_MATCH to atf-c++, which allows checking for a 318240116Smarcel regular expression match in a string. 319240116Smarcel 320240116Smarcel* Changed the default timeout for test cases from 5 minutes to 30 seconds. 321240116Smarcel 30 seconds is long enough for virtually all tests to complete, and 5 322240116Smarcel minutes is a way too long pause in a test suite where a single test case 323240116Smarcel stalls. 324240116Smarcel 325240116Smarcel* Deprecated the use.fs property. While this seemed like a good idea in 326240116Smarcel the first place to impose more control on what test cases can do, it 327240116Smarcel turns out to be bad. First, use.fs=false prevents bogus test cases 328240116Smarcel from dumping core so after-the-fact debugging is harder. Second, 329240116Smarcel supporting use.fs adds a lot of unnecessary complexity. atf-run will 330240116Smarcel now ignore any value provided to use.fs and will allow test cases to 331240116Smarcel freely access the file system if they wish to. 332240116Smarcel 333240116Smarcel* Added the atf_tc_get_config_var_as_{bool,long}{,_wd} functions to the atf-c 334240116Smarcel library. The 'text' module became private in 0.11 but was being used 335240116Smarcel externally to simplify the parsing of configuration variables. 336240116Smarcel 337240116Smarcel* Made atf-run recognize the 'unprivileged-user' configuration variable 338240116Smarcel and automatically drop root privileges when a test case sets 339240116Smarcel require.user=unprivileged. Note that this is, by no means, done for 340240116Smarcel security purposes; this is just for user convenience; tests should, in 341240116Smarcel general, not be blindly run as root in the first place. 342240116Smarcel 343240116Smarcel 344240116SmarcelChanges in version 0.11 345240116Smarcel*********************** 346240116Smarcel 347240116SmarcelExperimental version released on October 20th, 2010. 348240116Smarcel 349240116Smarcel* The ATF_CHECK* macros in atf-c++ were renamed to ATF_REQUIRE* to match 350240116Smarcel their counterparts in atf-c. 351240116Smarcel 352240116Smarcel* Clearly separated the modules in atf-c that are supposed to be public 353240116Smarcel from those that are implementation details. The header files for the 354240116Smarcel internal modules are not installed any more. 355240116Smarcel 356240116Smarcel* Made the atf-check tool private. It is only required by atf-sh and being 357240116Smarcel public has the danger of causing confusion. Also, making it private 358240116Smarcel simplifies the public API of atf. 359240116Smarcel 360240116Smarcel* Changed atf-sh to enable per-command error checking (set -e) by default. 361240116Smarcel This catches many cases in which a test case is broken but it is not 362240116Smarcel reported as such because execution continues. 363240116Smarcel 364240116Smarcel* Fixed the XSTL and CSS stylesheets to support expected failures. 365240116Smarcel 366240116Smarcel 367240116SmarcelChanges in version 0.10 368240116Smarcel*********************** 369240116Smarcel 370240116SmarcelExperimental version released on July 2nd, 2010. 371240116Smarcel 372240116SmarcelMiscellaneous features 373240116Smarcel 374240116Smarcel* Added expected failures support to test cases and atf-run. These 375240116Smarcel include, for example, expected clean exits, expected reception of fatal 376240116Smarcel signals, expected timeouts and expected errors in condition checks. 377240116Smarcel These statuses can be used to denote test cases that are known to fail 378240116Smarcel due to a bug in the code they are testing. atf-report reports these 379240116Smarcel tests separately but they do not count towards the failed test cases 380240116Smarcel amount. 381240116Smarcel 382240116Smarcel* Added the ATF_CHECK_ERRNO and ATF_REQUIRE_ERRNO to the C library to 383240116Smarcel allow easy checking of call failures that update errno. 384240116Smarcel 385240116Smarcel* Added the has.cleanup meta-data property to test caes that specifies 386240116Smarcel whether the test case has a cleanup routine or not; its value is 387240116Smarcel automatically set. This property is read by atf-run to know if it has to 388240116Smarcel run the cleanup routine; skipping this run for every test case 389240116Smarcel significantly speeds up the run time of test suites. 390240116Smarcel 391240116Smarcel* Reversed the order of the ATF_CHECK_THROW macro in the C++ binding to 392240116Smarcel take the expected exception as the first argument and the statement to 393240116Smarcel execute as the second argument. 394240116Smarcel 395240116SmarcelChanges in atf-check 396240116Smarcel 397240116Smarcel* Changed atf-check to support negating the status and output checks by 398240116Smarcel prefixing them with not- and added support to specify multiple checkers 399240116Smarcel for stdout and stderr, not only one. 400240116Smarcel 401240116Smarcel* Added the match output checker to atf-check to look for regular 402240116Smarcel expressions in the stdout and stderr of commands. 403240116Smarcel 404240116Smarcel* Modified the exit checks in atf-check to support checking for the 405240116Smarcel reception of signals. 406240116Smarcel 407240116SmarcelCode simplifications and cleanups 408240116Smarcel 409240116Smarcel* Removed usage messages from test programs to simplify the 410240116Smarcel implementation of every binding by a significant amount. They just now 411240116Smarcel refer the user to the appropriate manual page and do not attempt to wrap 412240116Smarcel lines on terminal boundaries. Test programs are not supposed to be run 413240116Smarcel by users directly so this minor interface regression is not important. 414240116Smarcel 415240116Smarcel* Removed the atf-format internal utility, which is unused after the 416240116Smarcel change documented above. 417240116Smarcel 418240116Smarcel* Removed the atf-cleanup internal utility. It has been unused since the 419240116Smarcel test case isolation was moved to atf-run in 0.8 420240116Smarcel 421240116Smarcel* Splitted the Makefile.am into smaller files for easier maintenance and 422240116Smarcel dropped the use of M4. Only affects users building from the repository 423240116Smarcel sources. 424240116Smarcel 425240116Smarcel* Intermixed tests with the source files in the source tree to provide 426240116Smarcel them more visibility and easier access. The tests directory is gone from 427240116Smarcel the source tree and tests are now suffixed by _test, not prefixed by t_. 428240116Smarcel 429240116Smarcel* Simplifications to the atf-c library: removed the io, tcr and ui 430240116Smarcel modules as they had become unnecessary after all simplifications 431240116Smarcel introduced since the 0.8 release. 432240116Smarcel 433240116Smarcel* Removed the application/X-atf-tcr format introduced in 0.8 release. 434240116Smarcel Tests now print a much simplified format that is easy to parse and nicer 435240116Smarcel to read by end users. As a side effect, the default for test cases is 436240116Smarcel now to print their results to stdout unless otherwise stated by providing 437240116Smarcel the -r flag. 438240116Smarcel 439240116Smarcel* Removed XML distribution documents and replaced them with plain-text 440240116Smarcel documents. They provided little value and introduced a lot of complexity 441240116Smarcel to the build system. 442240116Smarcel 443240116Smarcel* Simplified the output of atf-version by not attempting to print a 444240116Smarcel revision number when building form a distfile. Makes the build system 445240116Smarcel easier to maintain. 446240116Smarcel 447240116Smarcel 448240116SmarcelChanges in version 0.9 449240116Smarcel********************** 450240116Smarcel 451240116SmarcelExperimental version released on June 3rd, 2010. 452240116Smarcel 453240116Smarcel* Added atf-sh, an interpreter to process test programs written using 454240116Smarcel the shell API. This is not really a shell interpreter by itself though: 455240116Smarcel it is just a wrapper around the system shell that eases the loading of 456240116Smarcel the necessary ATF libraries. 457240116Smarcel 458240116Smarcel* Removed atf-compile in favour of atf-sh. 459240116Smarcel 460240116Smarcel* Added the use.fs metadata property to test case, which is used to 461240116Smarcel specify which test cases require file system access. This is to 462240116Smarcel highlight dependencies on external resources more clearly and to speed up 463240116Smarcel the execution of test suites by skipping the creation of many unnecessary 464240116Smarcel work directories. 465240116Smarcel 466240116Smarcel* Fixed test programs to get a sane default value for their source 467240116Smarcel directory. This means that it should not be necessary any more to pass 468240116Smarcel -s when running test programs that do not live in the current directory. 469240116Smarcel 470240116Smarcel* Defining test case headers became optional. This is trivial to achieve 471240116Smarcel in shell-based tests but a bit ugly in C and C++. In C, use the new 472240116Smarcel ATF_TC_WITHOUT_HEAD macro to define the test case, and in C++ use 473240116Smarcel ATF_TEST_CASE_WITHOUT_HEAD. 474240116Smarcel 475240116Smarcel 476240116SmarcelChanges in version 0.8 477240116Smarcel********************** 478240116Smarcel 479240116SmarcelExperimental version released on May 7th, 2010. 480240116Smarcel 481240116Smarcel* Test programs no longer run several test cases in a row. The execution 482240116Smarcel of a test program now requires a test case name, and that single test 483240116Smarcel case is executed. To execute several test cases, use the atf-run utility 484240116Smarcel as usual. 485240116Smarcel 486240116Smarcel* Test programs no longer fork a subprocess to isolate the execution of 487240116Smarcel test cases. They run the test case code in-process, and a crash of the 488240116Smarcel test case will result in a crash of the test program. This is to ease 489240116Smarcel debugging of faulty test cases. 490240116Smarcel 491240116Smarcel* Test programs no longer isolate their test cases. This means that they 492240116Smarcel will not create temporary directories nor sanitize the environment any 493240116Smarcel more. Yes: running a test case that depends on system state by hand will 494240116Smarcel most likely yield different results depending on where (machine, 495240116Smarcel directory, user environment, etc.) it is run. Isolation has been moved 496240116Smarcel to atf-run. 497240116Smarcel 498240116Smarcel* Test programs no longer print a cryptic format (application/X-atf-tcs) 499240116Smarcel on a special file channel. They can now print whatever they want on the 500240116Smarcel screen. Because test programs can now only run one test case every time, 501240116Smarcel providing controlled output is not necessary any more. 502240116Smarcel 503240116Smarcel* Test programs no longer write their status into a special file 504240116Smarcel descriptor. Instead, they create a file with the results, which is later 505240116Smarcel parsed by atf-run. This changes the semantics of the -r flag. 506240116Smarcel 507240116Smarcel* atf-run has been adjusted to perform the test case isolation. As a 508240116Smarcel result, there is now a single canonical place that implements the 509240116Smarcel isolation of test caes. In previous releases, the three language 510240116Smarcel bindings (C, C++ and shell) had to be kept in sync with each other (read: 511240116Smarcel not a nice thing to do at all). As a side effect of this change, writing 512240116Smarcel bindings for other languages will be much, much easier from now on. 513240116Smarcel 514240116Smarcel* atf-run forks test programs on a test case basis, instead of on a test 515240116Smarcel program basis as it did before. This is to provide the test case 516240116Smarcel isolation that was before implemented by the test programs themselves. 517240116Smarcel 518240116Smarcel* Removed the atf-exec tool. This was used to implement test case 519240116Smarcel isolation in atf-sh, but it is now unnecessary. 520240116Smarcel 521240116Smarcel* It is now optional to define the descr meta-data property. It has been 522240116Smarcel proven to be mostly useless, because test cases often carry a descriptive 523240116Smarcel name of their own. 524240116Smarcel 525240116Smarcel 526240116SmarcelChanges in version 0.7 527240116Smarcel********************** 528240116Smarcel 529240116SmarcelExperimental version released on December 22nd, 2009. 530240116Smarcel 531240116Smarcel* Added build-time checks to atf-c and atf-c++. A binding for atf-sh 532240116Smarcel will come later. 533240116Smarcel 534240116Smarcel* Migrated all build-time checks for header files to proper ATF tests. 535240116Smarcel This demonstrates the use of the new feature described above. 536240116Smarcel 537240116Smarcel* Added an internal API for child process management. 538240116Smarcel 539240116Smarcel* Converted all plain-text distribution documents to a Docbook canonical 540240116Smarcel version, and include pre-generated plain text and HTML copies in the 541240116Smarcel distribution file. 542240116Smarcel 543240116Smarcel* Simplified the contents of the Makefile.am by regenerating it from a 544240116Smarcel canonical Makefile.am.m4 source. As a side-effect, some dependency 545240116Smarcel specifications were fixed. 546240116Smarcel 547240116Smarcel* Migrated all checks from the check target to installcheck, as these 548240116Smarcel require ATF to be installed. 549240116Smarcel 550240116Smarcel* Fixed sign comparison mismatches triggered by the now-enabled 551240116Smarcel -Wsign-compare. 552240116Smarcel 553240116Smarcel* Fixed many memory and object leaks. 554240116Smarcel 555240116Smarcel 556240116SmarcelChanges in version 0.6 557240116Smarcel********************** 558240116Smarcel 559240116SmarcelExperimental version released on January 18th, 2009. 560240116Smarcel 561240116Smarcel* Make atf-exec be able to kill its child process after a certain period 562240116Smarcel of time; this is controlled through the new -t option. 563240116Smarcel 564240116Smarcel* Change atf-sh to use atf-exec's -t option to control the test case's 565240116Smarcel timeouts, instead of doing it internally. Same behavior as before, but 566240116Smarcel noticeably faster. 567240116Smarcel 568240116Smarcel* atf-exec's -g option and atf-killpg are gone due to the previous 569240116Smarcel change. 570240116Smarcel 571240116Smarcel* Added the atf-check(1) tool, a program that executes a given command 572240116Smarcel and checks its exit code against a known value and allows the management 573240116Smarcel of stdout and stderr in multiple ways. This replaces the previous 574240116Smarcel atf_check function in the atf-sh library and exposes this functionality 575240116Smarcel to both atf-c and atf-c++. 576240116Smarcel 577240116Smarcel* Added the ATF_REQUIRE family of macros to the C interface. These help 578240116Smarcel in checking for fatal test conditions. The old ATF_CHECK macros now 579240116Smarcel perform non-fatal checks only. I.e. by using ATF_CHECK, the test case 580240116Smarcel can now continue its execution and the failures will not be reported 581240116Smarcel until the end of the whole run. 582240116Smarcel 583240116Smarcel* Extended the amount of ATF_CHECK_* C macros with new ones to provide 584240116Smarcel more features to the developer. These also have their corresponding 585240116Smarcel counterparts in the ATF_REQUIRE_* family. The new macros (listing the 586240116Smarcel suffixes only) are: _EQ (replaces _EQUAL), _EQ_MSG, _STREQ and 587240116Smarcel _STREQ_MSG. 588240116Smarcel 589240116Smarcel 590240116SmarcelChanges in version 0.5 591240116Smarcel********************** 592240116Smarcel 593240116SmarcelExperimental version released on May 1st, 2008. 594240116Smarcel 595240116Smarcel* Clauses 3 and 4 of the BSD license used by the project were dropped. 596240116Smarcel All the code is now under a 2-clause BSD license compatible with the GNU 597240116Smarcel General Public License (GPL). 598240116Smarcel 599240116Smarcel* Added a C-only binding so that binary test programs do not need to be 600240116Smarcel tied to C++ at all. This binding is now known as the atf-c library. 601240116Smarcel 602240116Smarcel* Renamed the C++ binding to atf-c++ for consistency with the new atf-c. 603240116Smarcel 604240116Smarcel* Renamed the POSIX shell binding to atf-sh for consistency with the new 605240116Smarcel atf-c and atf-c++. 606240116Smarcel 607240116Smarcel* Added a -w flag to test programs through which it is possible to 608240116Smarcel specify the work directory to be used. This was possible in prior 609240116Smarcel releases by defining the workdir configuration variable (-v workdir=...), 610240116Smarcel but was a conceptually incorrect mechanism. 611240116Smarcel 612240116Smarcel* Test programs now preserve the execution order of test cases when they 613240116Smarcel are given in the command line. Even those mentioned more than once are 614240116Smarcel executed multiple times to comply with the user's requests. 615240116Smarcel 616240116Smarcel 617240116SmarcelChanges in version 0.4 618240116Smarcel********************** 619240116Smarcel 620240116SmarcelExperimental version released on February 4th, 2008. 621240116Smarcel 622240116Smarcel* Added two new manual pages, atf-c++-api and atf-sh-api, describing the 623240116Smarcel C++ and POSIX shell interfaces used to write test programs. 624240116Smarcel 625240116Smarcel* Added a pkg-config file, useful to get the flags to build against the 626240116Smarcel C++ library or to easily detect the presence of ATF. 627240116Smarcel 628240116Smarcel* Added a way for test cases to require a specific architecture and/or 629240116Smarcel machine type through the new 'require.arch' and 'require.machine' 630240116Smarcel meta-data properties, respectively. 631240116Smarcel 632240116Smarcel* Added the 'timeout' property to test cases, useful to set an 633240116Smarcel upper-bound limit for the test's run time and thus prevent global test 634240116Smarcel program stalls due to the test case's misbehavior. 635240116Smarcel 636240116Smarcel* Added the atf-exec(1) internal utility, used to execute a command 637240116Smarcel after changing the process group it belongs to. 638240116Smarcel 639240116Smarcel* Added the atf-killpg(1) internal utility, used to kill process groups. 640240116Smarcel 641240116Smarcel* Multiple portability fixes. Of special interest, full support for 642240116Smarcel SunOS (Solaris Express Developer Edition 2007/09) using the Sun Studio 12 643240116Smarcel C++ compiler. 644240116Smarcel 645240116Smarcel* Fixed a serious bug that prevented atf-run(1) from working at all 646240116Smarcel under Fedora 8 x86_64. Due to the nature of the bug, other platforms 647240116Smarcel were likely affected too. 648240116Smarcel 649240116Smarcel 650240116SmarcelChanges in version 0.3 651240116Smarcel********************** 652240116Smarcel 653240116SmarcelExperimental version released on November 11th, 2007. 654240116Smarcel 655240116Smarcel* Added XML output support to atf-report. This is accompanied by a DTD 656240116Smarcel for the format's structure and sample XSLT/CSS files to post-process this 657240116Smarcel output and convert it to a plain HTML report. 658240116Smarcel 659240116Smarcel* Changed atf-run to add system information to the report it generates. 660240116Smarcel This is currently used by atf-report's XML output only, and is later 661240116Smarcel printed in the HTML reports in a nice and useful summary table. The user 662240116Smarcel and system administrator are allowed to tune this feature by means of 663240116Smarcel hooks. 664240116Smarcel 665240116Smarcel* Removed the test cases' 'isolated' property. This was intended to 666240116Smarcel avoid touching the file system at all when running the related test case, 667240116Smarcel but this has not been true for a long while: some control files are 668240116Smarcel unconditionally required for several purposes, and we cannot easily get 669240116Smarcel rid of them. This way we remove several critical and delicate pieces of 670240116Smarcel code. 671240116Smarcel 672240116Smarcel* Improved atf-report's CSV output format to include information about 673240116Smarcel test programs too. 674240116Smarcel 675240116Smarcel* Fixed the tests that used atf-compile to not require this tool as a 676240116Smarcel helper. Avoids systems without build-time utilities to skip many tests 677240116Smarcel that could otherwise be run. (E.g. NetBSD without the comp.tgz set 678240116Smarcel installed.) 679240116Smarcel 680240116Smarcel* Many general cleanups: Fixed many pieces of code marked as ugly and/or 681240116Smarcel incomplete. 682240116Smarcel 683240116Smarcel 684240116SmarcelChanges in version 0.2 685240116Smarcel********************** 686240116Smarcel 687240116SmarcelExperimental version released on September 20th, 2007. 688240116Smarcel 689240116Smarcel* Test cases now get a known umask on entry. 690240116Smarcel 691240116Smarcel* atf-run now detects many unexpected failures caused by test programs and 692240116Smarcel reports them as bogus tests. atf-report is able to handle these new 693240116Smarcel errors and nicely reports them to the user. 694240116Smarcel 695240116Smarcel* All the data formats read and written by the tools have been 696240116Smarcel documented and cleaned up. These include those grammars that define how 697240116Smarcel the different components communicate with each other as well as the 698240116Smarcel format of files written by the developers and users: the Atffiles and the 699240116Smarcel configuration files. 700240116Smarcel 701240116Smarcel* Added the atf-version tool, a utility that displays information about 702240116Smarcel the currently installed version of ATF. 703240116Smarcel 704240116Smarcel* Test cases can now define an optional cleanup routine to undo their 705240116Smarcel actions regardless of their exit status. 706240116Smarcel 707240116Smarcel* atf-report now summarizes the list of failed (bogus) test programs 708240116Smarcel when using the ticker output format. 709240116Smarcel 710240116Smarcel* Test programs now capture some termination signals and clean up any 711240116Smarcel temporary files before exiting the program. 712240116Smarcel 713240116Smarcel* Multiple bug fixes and improvements all around. 714240116Smarcel 715240116Smarcel 716240116SmarcelChanges in version 0.1 717240116Smarcel********************** 718240116Smarcel 719240116SmarcelExperimental version released on August 20th, 2007. 720240116Smarcel 721240116Smarcel* First public version. This was released coinciding with the end of the 722240116Smarcel Google Summer of Code 2007 program. 723240116Smarcel 724240116Smarcel 725240116Smarcel=========================================================================== 726240116Smarcelvim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2 727