#
1.67 |
|
26-Dec-2022 |
jmc |
spelling fixes; from paul tagliamonte amendments to his diff are noted on tech
|
Revision tags: OPENBSD_7_1_BASE OPENBSD_7_2_BASE
|
#
1.66 |
|
13-Jan-2022 |
schwarze |
Only sort the result array if it contains more than one element, making the mansearch() function easier to read for human auditors. No functional change on OpenBSD.
As observed by Mark Millard <marklmi at yahoo dot com>, neither the latest version of POSIX 2008 nor C11 defines what qsort(3) should do for base == NULL && nmemb == 0. My impression is it is indeed undefined behaviour because the standards say that base shall point to an array, NULL does not point to an array, and while there is special wording saying that compar() shall not be called if nmemb == 0, i fail to see any similar wording stating that base shall not be accessed if nmemb == 0. Consequently, this patch is also likely to improve standard conformance and portability.
Minor issue found by Stefan Esser <se at FreeBSD> with UBSAN. He sent a patch to bugs@, but my patch differs in a minor way.
|
Revision tags: OPENBSD_6_6_BASE OPENBSD_6_7_BASE OPENBSD_6_8_BASE OPENBSD_6_9_BASE OPENBSD_7_0_BASE
|
#
1.65 |
|
01-Jul-2019 |
schwarze |
delete trailing whitespace and space-tab sequences; no code change; patch from Michal Nowak <mnowak at startmail dot com> who found these with git pbchk in the illumos tree
|
#
1.64 |
|
30-Apr-2019 |
schwarze |
In man(1) mode, i.e. when asking for a single manual page by name, prefer file name matches over .Dt/.TH matches over first NAME matches over later NAME matches, but do not change the ordering for apropos(1) nor for man -a.
This reverts main.c rev. 1.213 and mansearch.h rev. 1.23 and includes a partial revert of mansearch.c rev. 1.62.
Regression reported by Lorenzo Beretta <loreb at github> as part of https://github.com/void-linux/void-packages/issues/9868 .
|
Revision tags: OPENBSD_6_5_BASE
|
#
1.63 |
|
13-Dec-2018 |
schwarze |
Cleanup, no functional change: Split the top level parser interface out of the utility header mandoc.h, into a new header mandoc_parse.h, for use in the main program and in the main parser only. Move enum mandoc_os into roff.h because struct roff_man is the place where it is stored. This allows removal of mandoc.h from seven files in low-level parsers and in formatters.
|
#
1.62 |
|
22-Nov-2018 |
schwarze |
In apropos(1) output, stop sorting .Nm search results by name priorities (bits). The obscure feature wasn't documented and merely confused people - for example Edward Tomasz Napierala <trasz at FreeBSD>, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227408.
Smaller patch provided by Yuri Pankov <yuripv at FreeBSD>, but i'm also retiring the now unused "bits" member from struct manpage. Simplification is good.
|
#
1.61 |
|
19-Nov-2018 |
schwarze |
Improve POSIX compliance by making case-insensitive extended regular expressions the default in man(1) -k searches, also matching what the man-db package used by many Linux distributions does.
Originally requested by Wolfram Schneider <wosch at FreeBSD> via Yuri Pankov <yuripv at FreeBSD>.
Feedback and OK cheloha@, and no objections when shown on tech@.
Thanks to cheloha@ for pointing out that POSIX requires this behaviour and for the suggestion to explicitly say that *extended* regular expressions are used here.
While here, unify spelling of case-[in]sensitive, fix a typo, update the EXAMPLES, and add a STANDARDS section.
|
Revision tags: OPENBSD_6_2_BASE OPENBSD_6_3_BASE OPENBSD_6_4_BASE
|
#
1.60 |
|
22-Aug-2017 |
schwarze |
When finding a bogus database entry, print a meaningful warning and skip the entry. Issue reported by espie@.
|
#
1.59 |
|
02-Aug-2017 |
schwarze |
No longer use names that only occur in the SYNOPSIS section as names for man(1) lookup. For OpenBSD base and Xenocara, that functionality was never intended to be required, and i just fixed the last handful of offenders using it - not counting the horribly ill-designed interfaces engine(3) and lh_new(3) which are impossible to properly document in the first place.
Of course, apropos(1) and whatis(1) continue to use SYNOPSIS .Nm, .Fn, and .Fo macros, so "man -k ENGINE_get_load_privkey_function" still works.
This change also gets rid of a few bogus warnings "cross reference to self" which actually are *not* to self, like in yp(8).
This former functionality was intended to help third-party software in the ports tree and on non-OpenBSD systems containing manual pages with incomplete or corrupt NAME sections. But it turned out it did more harm than good, and caused more confusion than relief, specifically for third party manuals and for maintainers of mandoc-portable on other operating systems. So kill it. Problems reported, among others, by Yuri Pankov (illumos).
OK jmc@
|
#
1.58 |
|
19-Jul-2017 |
schwarze |
Prefer arch-dependent over arch-independent pages if the name priority, the section number, and all names match. Changes little on installed systems except the ordering of apropos(1) results, because we install base and Xenocara manuals in different trees, but fixes lookup of pages like apm(4) vs. apm(4/amd64) in man.cgi(8). Issue discovered by martian67 on freenode and reported via tj@.
|
#
1.57 |
|
01-Jul-2017 |
schwarze |
Basic reporting of .Xrs to manual pages that don't exist in the base system, inspired by mdoclint(1).
We are able to do this because (1) the -mdoc parser, the -Tlint validator, and the man(1) manual page lookup code are all in the same program and (2) the mandoc.db(5) database format allows fast lookup.
Feedback from, previous versions tested by, and OK jmc@.
A few features will be added to this in the tree, step by step.
|
#
1.56 |
|
17-May-2017 |
schwarze |
Do not warn if a database file does not exist because that is simply equivalent to an empty database. Suggested by ajacoutot@
|
#
1.55 |
|
18-Apr-2017 |
schwarze |
The apropos(1) manual still documents the unary -i operator for regular expression search terms, but it appears that somewhere along the way, the implementation got lost, so restore it. Bug found while investigating other reports from Gonzalo Tornaria.
|
#
1.54 |
|
18-Apr-2017 |
schwarze |
Fix previous: i forgot explicit NUL termination; noticed by Gonzalo <Tornaria at cmat dot edu dot uy>, thanks!
|
#
1.53 |
|
17-Apr-2017 |
schwarze |
Restore -kO Nd, -kO Nm, -kO sec, and -kO arch to working order. They got broken in the SQLite removal. As opposed to the rest of -kO, they are no longer very useful, but they are certainly not supposed to fail assertions. Issue reported by Gonzalo Tornaria <tornaria at cmat dot edu dot uy>.
|
Revision tags: OPENBSD_6_1_BASE
|
#
1.52 |
|
03-Mar-2017 |
schwarze |
A missing initialization could randomly cause regular expression searches to be case-insensitive that ought to be case sensitive. Found by jsg@ with scan-build.
|
#
1.51 |
|
01-Aug-2016 |
schwarze |
Remove the dependency on SQLite without loss of functionality. Drop the obsolete names_check() now that we deleted MLINKS. Run "doas makewhatis" after compiling and installing this.
Earlier version tested by jmc@ and jturner@; "commit it all" deraadt@ "commit and dodge" krw@
|
Revision tags: OPENBSD_6_0_BASE
|
#
1.50 |
|
09-Jul-2016 |
schwarze |
getopt(3) is declared in <unistd.h>, and <getopt.h> is not needed; from Joerg Sonnenberger via Thomas Klausner, NetBSD.
|
Revision tags: OPENBSD_5_9_BASE
|
#
1.49 |
|
08-Jan-2016 |
schwarze |
Prefer warn(3) over perror(3) at the few places where it was used. It is useful to see the program name. Suggested by Christos Zoulas (NetBSD).
|
#
1.48 |
|
26-Nov-2015 |
schwarze |
No point in trying to go on when elementary database operations like preparing queries or binding variables fail; that won't yield useful results anyway but may generate huge pointless error messages. Issue reported by deraadt@.
|
#
1.47 |
|
13-Oct-2015 |
schwarze |
Reduce the amount of code by moving the three copies of the ohash callback functions into one common place, preparing for the use of ohash for some additional purposes. No functional change.
|
#
1.46 |
|
11-Oct-2015 |
schwarze |
Finally use __progname, err(3) and warn(3). That's more readable and less error-prone than fumbling around with argv[0], fprintf(3), strerror(3), perror(3), and exit(3). It also shortens the code by 50 lines.
It's a bad idea to boycott good interfaces merely because standards committees ignore them. Instead, it's the job of the portable distribution to provide compatibility modules for archaic systems (like commercial Solaris) that still don't have them. Actually, the compat code for the portable distribution already exists and will be committed right after this.
|
#
1.45 |
|
06-Oct-2015 |
schwarze |
modernize style: "return" is not a function; ok cmp(1)
|
Revision tags: OPENBSD_5_8_BASE
|
#
1.44 |
|
01-Apr-2015 |
schwarze |
Let man(1) and apropos(1) work even when the current directory is unusable: Only change back to the current directory when the directory was changed before and the next path is relative. This is now more similar to what makewhatis(8) does. Issue reported by espie@.
|
#
1.43 |
|
27-Mar-2015 |
schwarze |
Parse the new man.conf(5) "output" directive. The next step will be to actually use the parsed data.
|
#
1.42 |
|
11-Mar-2015 |
schwarze |
When interpreting the -O argument as a macro name fails, fall back to showing Nd rather than not showing anything. Issue reported by jmc@.
|
Revision tags: OPENBSD_5_7_BASE
|
#
1.41 |
|
27-Feb-2015 |
schwarze |
When man(1) and apropos(1) look for a file man1/foo.1 but it's unavailable, fall back to glob(man1/foo.*), which is more like what old man(1) did. Do this both for file names from the database and for fs_lookup(). This is relevant because some ports install files like man1/xset.1x. Regression reported by patrick keshishian <pkeshish at gmail dot com>.
|
#
1.40 |
|
20-Jan-2015 |
schwarze |
Make the man(1) and apropos(1) options -s and -S much less expensive: Do not append an SQL clause looking into the large "keys" table. Instead, filter the result of the SQL query in buildnames() where equivalent data from the much smaller "mlinks" table is already available for free.
This is relevant because man(1) uses the equivalent of "-S ${MACHINE}" by default since main.c rev. 1.122, to make sure that manuals for the current architecture are shown. With many ports installed, this patch can speed up man(1) by a factor of more than a hundred.
Slowness reported by Theo Buehler <theo at math dot ethz dot ch>, thanks!
|
#
1.39 |
|
06-Dec-2014 |
schwarze |
When opening mandoc.db fails, tell the user in which directory. Improving an unhelpful error message reported by millert@.
|
#
1.38 |
|
27-Nov-2014 |
schwarze |
Make makewhatis(8) understand .so links to .gz pages. Drop the FORM_GZ annotation in the mpages table; it is conceptually wrong because it ought to be in the mlinks table: An uncompressed .so link file can point to a compressed manual page file and vice versa. Besides, it is no longer needed because mparse_open() handles it all. Sprinkle some KNF while here.
|
#
1.37 |
|
18-Nov-2014 |
schwarze |
In man(1) mode, prefer file name matches over .Dt name matches over first .Nm entries over other NAME .Nm entries over SYNOPSIS .Nm entries. For example, this makes sure "man ypbind" does not return yp(8). Re-run "makewhatis" to profit from this change.
|
#
1.36 |
|
11-Nov-2014 |
schwarze |
In man(1) mode without -a, stop searching after the first manual tree that contained at least one match in order to not prefer mdoc(1) from ports over mdoc(7). As a bonus, this results in a speedup.
|
#
1.35 |
|
03-Sep-2014 |
schwarze |
If a manual page is installed gzip(1)ed, let makewhatis(8) take note in mandoc.db(5), such that man(1) -w and apropos(1) -w can report the correct filename. This is a prerequisite for letting apropos -a and man support gzip'ed manuals in the future, which doesn't work yet.
|
#
1.34 |
|
01-Sep-2014 |
schwarze |
In man(1) mode, change to the right directory before starting the parser, just like traditional man(1) does, such that .so links have a chance to work. After this point, we don't need the current directory for anything else before exit, so we don't need to worry about getting back and we can safely ignore failure. This lets man(1) find more Xenocara manuals, but not all of them yet. Other issues remain that need to be fixed, too.
|
#
1.33 |
|
27-Aug-2014 |
schwarze |
Add an implementation of man(1) into the /usr/bin/mandoc binary and provide a unified set of command line options for mandoc(1), man(1), apropos(1), and whatis(1), each option doing the same for all four. Not adding any completely new options, only extending exiting ones from one tool to the others. New options are: * apropos & whatis -acfkw (in the past, these were man(1) only) * apropos & whatis -a -IOTW (in the past, mandoc(1) only) * mandoc -ac (in the past, man(1) only) * man -IOTW (in the past, mandoc(1) only)
Before we can decide whether or not we want to replace src/usr.bin/man with this implementation, considerable bugfixing, testing, and performance measurements are needed, which i'd rather do in the tree than outside. Note that these bugs only affect the new man(1) mode, existing mandoc(1), apropos(1), and whatis(1) is fine.
The new functionality in mandoc(1), apropos(1), and whatis(1) is fully enabled. To play with the new man(1), you can try: # mv /usr/bin/man /usr/bin/oman # ln -s /usr/bin/mandoc /usr/bin/man
Positive feedback about the general direction from sthen@ and jmc@, and deraadt@ is not against it.
|
#
1.32 |
|
21-Aug-2014 |
schwarze |
Bugfix: make whatis(1) case-insensitive again. The traditional whatis(1) was case-insensitve and it's still documented that way, that but apparently got broken with or after the switch.
|
#
1.31 |
|
09-Aug-2014 |
schwarze |
mmap(2) requires MAP_PRIVATE ^ MAP_SHARED for flags; found by kristaps@ on Mac OS X
|
#
1.30 |
|
08-Aug-2014 |
schwarze |
Absurdly, the return value of sqlite3_column_text() is "const unsigned char *", which causes warnings with GCC on Linux. Explicitly cast to "const char *" to avoid this. Issue noticed by kristaps@.
|
Revision tags: OPENBSD_5_6_BASE
|
#
1.29 |
|
24-Jul-2014 |
schwarze |
Sort result pages first by section number, then by name. By moving the sort from cgi.c to mansearch.c, we get two advantages: Easier access to the data needed for sorting, in particular the section number, and the apropos(1) command line utility profits as well.
Feature requested by deraadt@.
|
#
1.28 |
|
12-Jul-2014 |
schwarze |
Fix whatis(1) to correctly match words instead of any substrings. While here, also provide an internal mode (MANSEARCH_MAN) to match complete names, to be used by man.cgi(8).
|
#
1.27 |
|
12-May-2014 |
espie |
adjust to ohash being in libutil now, and to the interface changes. fix potential integer overflows in memory allocation (mostly for pedagogical purposes, these are unlikely to overflow in practice) move the rest of lst.lib stuff into its own directory.
|
#
1.26 |
|
23-Apr-2014 |
schwarze |
Audit malloc(3)/calloc(3)/realloc(3) usage. * Change eight reallocs to reallocarray to be safe from overflows. * Change one malloc to reallocarray to be safe from overflows. * Change one calloc to reallocarray, no zeroing needed. * Change the order of arguments of three callocs (aesthetical).
|
#
1.25 |
|
23-Apr-2014 |
schwarze |
improve SQL style: avoid "SELECT *", be explicit in what columns we want; suggested by espie@.
|
#
1.24 |
|
20-Apr-2014 |
schwarze |
KNF: case (FOO): -> case FOO, remove /* LINTED */ and /* ARGSUSED */, remove trailing whitespace and blanks before tabs, improve some indenting; no functional change
|
#
1.23 |
|
17-Apr-2014 |
schwarze |
Garbage collect one pair of needless parentheses in SQL code generation; note this doesn't affect performance, SQLite generates the same byte code. While here, make the calls to exprspec() easier to understand.
|
#
1.22 |
|
16-Apr-2014 |
schwarze |
Rename the mpages.id column to mpages.pageid. There is no good reason to call this kid by a different name here than in all other tables. Easier to polish this now than after enabling.
|
#
1.21 |
|
16-Apr-2014 |
schwarze |
Pass the function flags SQLITE_UTF8 (because SQLITE_ANY is deprecated) and SQLITE_DETERMINISTIC when creating deterministic functions; best practice measure suggested by espie@ and jeremy@; as expected by jeremy@, no measurable effect on performance.
|
#
1.20 |
|
15-Apr-2014 |
schwarze |
At the end of mansearch(), fchdir() back to where we started from; this is cleaner and helps to not scatter gmon.out files all over the place when profiling.
|
#
1.19 |
|
11-Apr-2014 |
schwarze |
Further apropos(1) speed optimization was trickier than anticipated. Contrary to what i initially thought, almost all time is now spent inside sqlite3(3) routines, and i found no easy way calling less of them. However, sqlite(3) spends substantial time in malloc(3), and even more (twice that) in its immediate malloc wrapper, sqlite3MemMalloc(), keeping track of all individual malloc chunk sizes. Typically about 90% of the malloced memory is used for purposes of the pagecache.
By providing an mmap(3) MAP_ANON SQLITE_CONFIG_PAGECACHE, execution time decreases by 20-25% for simple (Nd and/or Nm) queries, 10-20% for non-NAME queries, and even apropos(1) resident memory size as reported by top(1) decreases by 20% for simple and by 60% for non-NAME queries. The new function, mansearch_setup(), spends no measurable time.
The pagesize chosen is optimal: * Substantially smaller pages yield no gain at all. * Larger pages provide no additional benefit and just waste memory.
The chosen number of pages in the cache is a compromise: * For simple queries, a handful of pages would suffice to get the full speed effect, at an apropos(1) resident memory size of about 2.0 MB. * For non-NAME queries, a large pagecache with 2k pages (2.5 MB) might gain a few more percent in speed, but at the expense of doubling the apropos(1) resident memory size for *all* queries. * The chosen number of 256 pages (330 kB) allows nearly full speed gain for all queries at the price of a 15% resident memory size increase.
|
#
1.18 |
|
10-Apr-2014 |
schwarze |
Next speed optimization step for the new apropos(1). Split manual names out of the common "keys" table into their own "names" table. This reduces standard apropos(1) search times (i.e. searching for names and descriptions only) by typically about 70% for the full /usr/share/man database. (Yes, that multiplies with the previous optimization step, so both together have reduced search times by a factor of more than six. I'm not done yet, expect more to come.)
Even with the minimal databases built with makewhatis(8) -Q, this step still reduces search times by 15-20%. For both cases, database sizes and build times hardly change (+/-2%).
|
#
1.17 |
|
09-Apr-2014 |
schwarze |
After careful gprof(1)ing of the new apropos(1), move the descriptions back from the keys table to the mpages table: I found a good way to still use them in searches, without complication of the code.
On my notebook, this reduces typical apropos(1) search times by about 40%, it reduces /usr/share/man database size by 6% in makewhatis(8) -Q mode and by 2% in standard mode (less overhead storing pointers to mpages), and it doesn't measurably change database build times (may even be going down by a percent or so because less data is being copied around in ohashes).
|
#
1.16 |
|
28-Mar-2014 |
schwarze |
Properly initialize malloc(3)ed memory. With this bug fix, partly unitialized memory could sometimes be returned, sometimes causing crashes by bogus free(3)s in apropos(1).
|
#
1.15 |
|
21-Mar-2014 |
schwarze |
avoid repetitive code for asprintf error handling
|
#
1.14 |
|
21-Mar-2014 |
schwarze |
The files mandoc.c and mandoc.h contained both specialised low-level functions used for multiple languages (mdoc, man, roff), for example mandoc_escape(), mandoc_getarg(), mandoc_eos(), and generic auxiliary functions. Split the auxiliaries out into their own file and header. While here, do some #include cleanup.
|
#
1.13 |
|
17-Mar-2014 |
schwarze |
in apropos(1) output, sort names and avoid multiple section numbers
|
Revision tags: OPENBSD_5_5_BASE
|
#
1.12 |
|
19-Jan-2014 |
schwarze |
Always compare arch case-insensitively.
|
#
1.11 |
|
19-Jan-2014 |
schwarze |
Get rid of the local keys table, use the new mansearch_const.c. No functional change.
|
#
1.10 |
|
06-Jan-2014 |
schwarze |
Remove the redundant "file" column from the "mlinks" table. The contents can easily be reconstructed from sec, arch, name, form. Shrinks the database by another 3% in standard mode and 9% in -Q mode.
|
#
1.9 |
|
06-Jan-2014 |
schwarze |
Drop Nd from the mpages table, it is still in the keys table. This shrinks the database in standard mode by 3%, in -Q mode by 9%, without loss of functionality.
|
#
1.8 |
|
05-Jan-2014 |
schwarze |
Remove the obsolete file name column from the mpages table. This column wasn't helpful because one manpage can have multiple MLINKS. Use the file name column in the mlinks table, instead.
|
#
1.7 |
|
05-Jan-2014 |
schwarze |
Remove the obsolete sec and arch columns from the mpages table. They were confusing because a manpage can have MLINKS in different sections and architectures.
|
#
1.6 |
|
05-Jan-2014 |
schwarze |
Reimplement apropos -s NUM -S ARCH EXPR by internally converting it to apropos \( EXPR \) -a 'sec~^NUM$' -a 'arch~^(ARCH|any)$' in preparation for removal of sec and arch from the mpage table.
Almost no functional change except for the following bonus: This also makes sure that for cross-section and cross-arch MLINKs, all of the following work: apropos -s 1 encrypt apropos -s 8 encrypt apropos -s 1 makekey apropos -s 8 makekey Before this commit, they don't, neither for the old makewhatis(8)/apropos(1) combo nor for the new one.
While here, print error messages about invalid regexps to stderr.
|
#
1.5 |
|
05-Jan-2014 |
schwarze |
Put section and architecture info into the keys table, in preparation for removing them from the mpages table, aiming for cleaner and more uniform interfaces. Database growth is below 4%, part of which will be reclaimed.
As a bonus, this allows searches like: ./obj/apropos An=kettenis -a arch=ppc ./obj/apropos An=kettenis -a sec~[^4]
|
#
1.4 |
|
04-Jan-2014 |
schwarze |
New implementation of complex search criteria using \(, \), -a because the old implementation got lost in the Berkeley to SQLite switch. Note that this is not just feature creep, but required for upcoming database format cleanup and simplification.
The following takes less than half a second on my Z61m ThinkPad: time ./obj/apropos An=kettenis -a \( Cd=vnet -o Xr=vbus \)
|
#
1.3 |
|
31-Dec-2013 |
schwarze |
Experimental feature to let apropos(1) show different keys than .Nd. This really takes us beyond what grep -R /usr/*/man/ can do because now you can search for pages by *one* criterion and then display the contents of *another* macro from those pages, like in $ apropos -O Ox Fa~wchar to get an impression how long wide character handling is available.
|
#
1.2 |
|
31-Dec-2013 |
schwarze |
Split buildnames() out of mansearch(); the latter function is getting too long and unwieldy, but will grow more code soon. No functional change.
|
#
1.1 |
|
31-Dec-2013 |
schwarze |
Replace the Berkeley-DB based mandocdb(8) by an SQLite3-based version, aiming for more flexible development and optimization options. Kristaps started this during the summer 2012, i did some very heavy bugfixing during t2k13 and finally, during the last few days, got it to a state where it is ripe for in-tree development. Beware, neither the user interfaces nor the database formats are expected to be stable just yet.
Will not be installed or activated until further discussion. No functional change to mandoc(1).
"As long as it remains off until we decide the cost, fine." deraadt@
|
#
1.66 |
|
13-Jan-2022 |
schwarze |
Only sort the result array if it contains more than one element, making the mansearch() function easier to read for human auditors. No functional change on OpenBSD.
As observed by Mark Millard <marklmi at yahoo dot com>, neither the latest version of POSIX 2008 nor C11 defines what qsort(3) should do for base == NULL && nmemb == 0. My impression is it is indeed undefined behaviour because the standards say that base shall point to an array, NULL does not point to an array, and while there is special wording saying that compar() shall not be called if nmemb == 0, i fail to see any similar wording stating that base shall not be accessed if nmemb == 0. Consequently, this patch is also likely to improve standard conformance and portability.
Minor issue found by Stefan Esser <se at FreeBSD> with UBSAN. He sent a patch to bugs@, but my patch differs in a minor way.
|
Revision tags: OPENBSD_6_6_BASE OPENBSD_6_7_BASE OPENBSD_6_8_BASE OPENBSD_6_9_BASE OPENBSD_7_0_BASE
|
#
1.65 |
|
01-Jul-2019 |
schwarze |
delete trailing whitespace and space-tab sequences; no code change; patch from Michal Nowak <mnowak at startmail dot com> who found these with git pbchk in the illumos tree
|
#
1.64 |
|
30-Apr-2019 |
schwarze |
In man(1) mode, i.e. when asking for a single manual page by name, prefer file name matches over .Dt/.TH matches over first NAME matches over later NAME matches, but do not change the ordering for apropos(1) nor for man -a.
This reverts main.c rev. 1.213 and mansearch.h rev. 1.23 and includes a partial revert of mansearch.c rev. 1.62.
Regression reported by Lorenzo Beretta <loreb at github> as part of https://github.com/void-linux/void-packages/issues/9868 .
|
Revision tags: OPENBSD_6_5_BASE
|
#
1.63 |
|
13-Dec-2018 |
schwarze |
Cleanup, no functional change: Split the top level parser interface out of the utility header mandoc.h, into a new header mandoc_parse.h, for use in the main program and in the main parser only. Move enum mandoc_os into roff.h because struct roff_man is the place where it is stored. This allows removal of mandoc.h from seven files in low-level parsers and in formatters.
|
#
1.62 |
|
22-Nov-2018 |
schwarze |
In apropos(1) output, stop sorting .Nm search results by name priorities (bits). The obscure feature wasn't documented and merely confused people - for example Edward Tomasz Napierala <trasz at FreeBSD>, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227408.
Smaller patch provided by Yuri Pankov <yuripv at FreeBSD>, but i'm also retiring the now unused "bits" member from struct manpage. Simplification is good.
|
#
1.61 |
|
19-Nov-2018 |
schwarze |
Improve POSIX compliance by making case-insensitive extended regular expressions the default in man(1) -k searches, also matching what the man-db package used by many Linux distributions does.
Originally requested by Wolfram Schneider <wosch at FreeBSD> via Yuri Pankov <yuripv at FreeBSD>.
Feedback and OK cheloha@, and no objections when shown on tech@.
Thanks to cheloha@ for pointing out that POSIX requires this behaviour and for the suggestion to explicitly say that *extended* regular expressions are used here.
While here, unify spelling of case-[in]sensitive, fix a typo, update the EXAMPLES, and add a STANDARDS section.
|
Revision tags: OPENBSD_6_2_BASE OPENBSD_6_3_BASE OPENBSD_6_4_BASE
|
#
1.60 |
|
22-Aug-2017 |
schwarze |
When finding a bogus database entry, print a meaningful warning and skip the entry. Issue reported by espie@.
|
#
1.59 |
|
02-Aug-2017 |
schwarze |
No longer use names that only occur in the SYNOPSIS section as names for man(1) lookup. For OpenBSD base and Xenocara, that functionality was never intended to be required, and i just fixed the last handful of offenders using it - not counting the horribly ill-designed interfaces engine(3) and lh_new(3) which are impossible to properly document in the first place.
Of course, apropos(1) and whatis(1) continue to use SYNOPSIS .Nm, .Fn, and .Fo macros, so "man -k ENGINE_get_load_privkey_function" still works.
This change also gets rid of a few bogus warnings "cross reference to self" which actually are *not* to self, like in yp(8).
This former functionality was intended to help third-party software in the ports tree and on non-OpenBSD systems containing manual pages with incomplete or corrupt NAME sections. But it turned out it did more harm than good, and caused more confusion than relief, specifically for third party manuals and for maintainers of mandoc-portable on other operating systems. So kill it. Problems reported, among others, by Yuri Pankov (illumos).
OK jmc@
|
#
1.58 |
|
19-Jul-2017 |
schwarze |
Prefer arch-dependent over arch-independent pages if the name priority, the section number, and all names match. Changes little on installed systems except the ordering of apropos(1) results, because we install base and Xenocara manuals in different trees, but fixes lookup of pages like apm(4) vs. apm(4/amd64) in man.cgi(8). Issue discovered by martian67 on freenode and reported via tj@.
|
#
1.57 |
|
01-Jul-2017 |
schwarze |
Basic reporting of .Xrs to manual pages that don't exist in the base system, inspired by mdoclint(1).
We are able to do this because (1) the -mdoc parser, the -Tlint validator, and the man(1) manual page lookup code are all in the same program and (2) the mandoc.db(5) database format allows fast lookup.
Feedback from, previous versions tested by, and OK jmc@.
A few features will be added to this in the tree, step by step.
|
#
1.56 |
|
17-May-2017 |
schwarze |
Do not warn if a database file does not exist because that is simply equivalent to an empty database. Suggested by ajacoutot@
|
#
1.55 |
|
18-Apr-2017 |
schwarze |
The apropos(1) manual still documents the unary -i operator for regular expression search terms, but it appears that somewhere along the way, the implementation got lost, so restore it. Bug found while investigating other reports from Gonzalo Tornaria.
|
#
1.54 |
|
18-Apr-2017 |
schwarze |
Fix previous: i forgot explicit NUL termination; noticed by Gonzalo <Tornaria at cmat dot edu dot uy>, thanks!
|
#
1.53 |
|
17-Apr-2017 |
schwarze |
Restore -kO Nd, -kO Nm, -kO sec, and -kO arch to working order. They got broken in the SQLite removal. As opposed to the rest of -kO, they are no longer very useful, but they are certainly not supposed to fail assertions. Issue reported by Gonzalo Tornaria <tornaria at cmat dot edu dot uy>.
|
Revision tags: OPENBSD_6_1_BASE
|
#
1.52 |
|
03-Mar-2017 |
schwarze |
A missing initialization could randomly cause regular expression searches to be case-insensitive that ought to be case sensitive. Found by jsg@ with scan-build.
|
#
1.51 |
|
01-Aug-2016 |
schwarze |
Remove the dependency on SQLite without loss of functionality. Drop the obsolete names_check() now that we deleted MLINKS. Run "doas makewhatis" after compiling and installing this.
Earlier version tested by jmc@ and jturner@; "commit it all" deraadt@ "commit and dodge" krw@
|
Revision tags: OPENBSD_6_0_BASE
|
#
1.50 |
|
09-Jul-2016 |
schwarze |
getopt(3) is declared in <unistd.h>, and <getopt.h> is not needed; from Joerg Sonnenberger via Thomas Klausner, NetBSD.
|
Revision tags: OPENBSD_5_9_BASE
|
#
1.49 |
|
08-Jan-2016 |
schwarze |
Prefer warn(3) over perror(3) at the few places where it was used. It is useful to see the program name. Suggested by Christos Zoulas (NetBSD).
|
#
1.48 |
|
26-Nov-2015 |
schwarze |
No point in trying to go on when elementary database operations like preparing queries or binding variables fail; that won't yield useful results anyway but may generate huge pointless error messages. Issue reported by deraadt@.
|
#
1.47 |
|
13-Oct-2015 |
schwarze |
Reduce the amount of code by moving the three copies of the ohash callback functions into one common place, preparing for the use of ohash for some additional purposes. No functional change.
|
#
1.46 |
|
11-Oct-2015 |
schwarze |
Finally use __progname, err(3) and warn(3). That's more readable and less error-prone than fumbling around with argv[0], fprintf(3), strerror(3), perror(3), and exit(3). It also shortens the code by 50 lines.
It's a bad idea to boycott good interfaces merely because standards committees ignore them. Instead, it's the job of the portable distribution to provide compatibility modules for archaic systems (like commercial Solaris) that still don't have them. Actually, the compat code for the portable distribution already exists and will be committed right after this.
|
#
1.45 |
|
06-Oct-2015 |
schwarze |
modernize style: "return" is not a function; ok cmp(1)
|
Revision tags: OPENBSD_5_8_BASE
|
#
1.44 |
|
01-Apr-2015 |
schwarze |
Let man(1) and apropos(1) work even when the current directory is unusable: Only change back to the current directory when the directory was changed before and the next path is relative. This is now more similar to what makewhatis(8) does. Issue reported by espie@.
|
#
1.43 |
|
27-Mar-2015 |
schwarze |
Parse the new man.conf(5) "output" directive. The next step will be to actually use the parsed data.
|
#
1.42 |
|
11-Mar-2015 |
schwarze |
When interpreting the -O argument as a macro name fails, fall back to showing Nd rather than not showing anything. Issue reported by jmc@.
|
Revision tags: OPENBSD_5_7_BASE
|
#
1.41 |
|
27-Feb-2015 |
schwarze |
When man(1) and apropos(1) look for a file man1/foo.1 but it's unavailable, fall back to glob(man1/foo.*), which is more like what old man(1) did. Do this both for file names from the database and for fs_lookup(). This is relevant because some ports install files like man1/xset.1x. Regression reported by patrick keshishian <pkeshish at gmail dot com>.
|
#
1.40 |
|
20-Jan-2015 |
schwarze |
Make the man(1) and apropos(1) options -s and -S much less expensive: Do not append an SQL clause looking into the large "keys" table. Instead, filter the result of the SQL query in buildnames() where equivalent data from the much smaller "mlinks" table is already available for free.
This is relevant because man(1) uses the equivalent of "-S ${MACHINE}" by default since main.c rev. 1.122, to make sure that manuals for the current architecture are shown. With many ports installed, this patch can speed up man(1) by a factor of more than a hundred.
Slowness reported by Theo Buehler <theo at math dot ethz dot ch>, thanks!
|
#
1.39 |
|
06-Dec-2014 |
schwarze |
When opening mandoc.db fails, tell the user in which directory. Improving an unhelpful error message reported by millert@.
|
#
1.38 |
|
27-Nov-2014 |
schwarze |
Make makewhatis(8) understand .so links to .gz pages. Drop the FORM_GZ annotation in the mpages table; it is conceptually wrong because it ought to be in the mlinks table: An uncompressed .so link file can point to a compressed manual page file and vice versa. Besides, it is no longer needed because mparse_open() handles it all. Sprinkle some KNF while here.
|
#
1.37 |
|
18-Nov-2014 |
schwarze |
In man(1) mode, prefer file name matches over .Dt name matches over first .Nm entries over other NAME .Nm entries over SYNOPSIS .Nm entries. For example, this makes sure "man ypbind" does not return yp(8). Re-run "makewhatis" to profit from this change.
|
#
1.36 |
|
11-Nov-2014 |
schwarze |
In man(1) mode without -a, stop searching after the first manual tree that contained at least one match in order to not prefer mdoc(1) from ports over mdoc(7). As a bonus, this results in a speedup.
|
#
1.35 |
|
03-Sep-2014 |
schwarze |
If a manual page is installed gzip(1)ed, let makewhatis(8) take note in mandoc.db(5), such that man(1) -w and apropos(1) -w can report the correct filename. This is a prerequisite for letting apropos -a and man support gzip'ed manuals in the future, which doesn't work yet.
|
#
1.34 |
|
01-Sep-2014 |
schwarze |
In man(1) mode, change to the right directory before starting the parser, just like traditional man(1) does, such that .so links have a chance to work. After this point, we don't need the current directory for anything else before exit, so we don't need to worry about getting back and we can safely ignore failure. This lets man(1) find more Xenocara manuals, but not all of them yet. Other issues remain that need to be fixed, too.
|
#
1.33 |
|
27-Aug-2014 |
schwarze |
Add an implementation of man(1) into the /usr/bin/mandoc binary and provide a unified set of command line options for mandoc(1), man(1), apropos(1), and whatis(1), each option doing the same for all four. Not adding any completely new options, only extending exiting ones from one tool to the others. New options are: * apropos & whatis -acfkw (in the past, these were man(1) only) * apropos & whatis -a -IOTW (in the past, mandoc(1) only) * mandoc -ac (in the past, man(1) only) * man -IOTW (in the past, mandoc(1) only)
Before we can decide whether or not we want to replace src/usr.bin/man with this implementation, considerable bugfixing, testing, and performance measurements are needed, which i'd rather do in the tree than outside. Note that these bugs only affect the new man(1) mode, existing mandoc(1), apropos(1), and whatis(1) is fine.
The new functionality in mandoc(1), apropos(1), and whatis(1) is fully enabled. To play with the new man(1), you can try: # mv /usr/bin/man /usr/bin/oman # ln -s /usr/bin/mandoc /usr/bin/man
Positive feedback about the general direction from sthen@ and jmc@, and deraadt@ is not against it.
|
#
1.32 |
|
21-Aug-2014 |
schwarze |
Bugfix: make whatis(1) case-insensitive again. The traditional whatis(1) was case-insensitve and it's still documented that way, that but apparently got broken with or after the switch.
|
#
1.31 |
|
09-Aug-2014 |
schwarze |
mmap(2) requires MAP_PRIVATE ^ MAP_SHARED for flags; found by kristaps@ on Mac OS X
|
#
1.30 |
|
08-Aug-2014 |
schwarze |
Absurdly, the return value of sqlite3_column_text() is "const unsigned char *", which causes warnings with GCC on Linux. Explicitly cast to "const char *" to avoid this. Issue noticed by kristaps@.
|
Revision tags: OPENBSD_5_6_BASE
|
#
1.29 |
|
24-Jul-2014 |
schwarze |
Sort result pages first by section number, then by name. By moving the sort from cgi.c to mansearch.c, we get two advantages: Easier access to the data needed for sorting, in particular the section number, and the apropos(1) command line utility profits as well.
Feature requested by deraadt@.
|
#
1.28 |
|
12-Jul-2014 |
schwarze |
Fix whatis(1) to correctly match words instead of any substrings. While here, also provide an internal mode (MANSEARCH_MAN) to match complete names, to be used by man.cgi(8).
|
#
1.27 |
|
12-May-2014 |
espie |
adjust to ohash being in libutil now, and to the interface changes. fix potential integer overflows in memory allocation (mostly for pedagogical purposes, these are unlikely to overflow in practice) move the rest of lst.lib stuff into its own directory.
|
#
1.26 |
|
23-Apr-2014 |
schwarze |
Audit malloc(3)/calloc(3)/realloc(3) usage. * Change eight reallocs to reallocarray to be safe from overflows. * Change one malloc to reallocarray to be safe from overflows. * Change one calloc to reallocarray, no zeroing needed. * Change the order of arguments of three callocs (aesthetical).
|
#
1.25 |
|
23-Apr-2014 |
schwarze |
improve SQL style: avoid "SELECT *", be explicit in what columns we want; suggested by espie@.
|
#
1.24 |
|
20-Apr-2014 |
schwarze |
KNF: case (FOO): -> case FOO, remove /* LINTED */ and /* ARGSUSED */, remove trailing whitespace and blanks before tabs, improve some indenting; no functional change
|
#
1.23 |
|
17-Apr-2014 |
schwarze |
Garbage collect one pair of needless parentheses in SQL code generation; note this doesn't affect performance, SQLite generates the same byte code. While here, make the calls to exprspec() easier to understand.
|
#
1.22 |
|
16-Apr-2014 |
schwarze |
Rename the mpages.id column to mpages.pageid. There is no good reason to call this kid by a different name here than in all other tables. Easier to polish this now than after enabling.
|
#
1.21 |
|
16-Apr-2014 |
schwarze |
Pass the function flags SQLITE_UTF8 (because SQLITE_ANY is deprecated) and SQLITE_DETERMINISTIC when creating deterministic functions; best practice measure suggested by espie@ and jeremy@; as expected by jeremy@, no measurable effect on performance.
|
#
1.20 |
|
15-Apr-2014 |
schwarze |
At the end of mansearch(), fchdir() back to where we started from; this is cleaner and helps to not scatter gmon.out files all over the place when profiling.
|
#
1.19 |
|
11-Apr-2014 |
schwarze |
Further apropos(1) speed optimization was trickier than anticipated. Contrary to what i initially thought, almost all time is now spent inside sqlite3(3) routines, and i found no easy way calling less of them. However, sqlite(3) spends substantial time in malloc(3), and even more (twice that) in its immediate malloc wrapper, sqlite3MemMalloc(), keeping track of all individual malloc chunk sizes. Typically about 90% of the malloced memory is used for purposes of the pagecache.
By providing an mmap(3) MAP_ANON SQLITE_CONFIG_PAGECACHE, execution time decreases by 20-25% for simple (Nd and/or Nm) queries, 10-20% for non-NAME queries, and even apropos(1) resident memory size as reported by top(1) decreases by 20% for simple and by 60% for non-NAME queries. The new function, mansearch_setup(), spends no measurable time.
The pagesize chosen is optimal: * Substantially smaller pages yield no gain at all. * Larger pages provide no additional benefit and just waste memory.
The chosen number of pages in the cache is a compromise: * For simple queries, a handful of pages would suffice to get the full speed effect, at an apropos(1) resident memory size of about 2.0 MB. * For non-NAME queries, a large pagecache with 2k pages (2.5 MB) might gain a few more percent in speed, but at the expense of doubling the apropos(1) resident memory size for *all* queries. * The chosen number of 256 pages (330 kB) allows nearly full speed gain for all queries at the price of a 15% resident memory size increase.
|
#
1.18 |
|
10-Apr-2014 |
schwarze |
Next speed optimization step for the new apropos(1). Split manual names out of the common "keys" table into their own "names" table. This reduces standard apropos(1) search times (i.e. searching for names and descriptions only) by typically about 70% for the full /usr/share/man database. (Yes, that multiplies with the previous optimization step, so both together have reduced search times by a factor of more than six. I'm not done yet, expect more to come.)
Even with the minimal databases built with makewhatis(8) -Q, this step still reduces search times by 15-20%. For both cases, database sizes and build times hardly change (+/-2%).
|
#
1.17 |
|
09-Apr-2014 |
schwarze |
After careful gprof(1)ing of the new apropos(1), move the descriptions back from the keys table to the mpages table: I found a good way to still use them in searches, without complication of the code.
On my notebook, this reduces typical apropos(1) search times by about 40%, it reduces /usr/share/man database size by 6% in makewhatis(8) -Q mode and by 2% in standard mode (less overhead storing pointers to mpages), and it doesn't measurably change database build times (may even be going down by a percent or so because less data is being copied around in ohashes).
|
#
1.16 |
|
28-Mar-2014 |
schwarze |
Properly initialize malloc(3)ed memory. With this bug fix, partly unitialized memory could sometimes be returned, sometimes causing crashes by bogus free(3)s in apropos(1).
|
#
1.15 |
|
21-Mar-2014 |
schwarze |
avoid repetitive code for asprintf error handling
|
#
1.14 |
|
21-Mar-2014 |
schwarze |
The files mandoc.c and mandoc.h contained both specialised low-level functions used for multiple languages (mdoc, man, roff), for example mandoc_escape(), mandoc_getarg(), mandoc_eos(), and generic auxiliary functions. Split the auxiliaries out into their own file and header. While here, do some #include cleanup.
|
#
1.13 |
|
17-Mar-2014 |
schwarze |
in apropos(1) output, sort names and avoid multiple section numbers
|
Revision tags: OPENBSD_5_5_BASE
|
#
1.12 |
|
19-Jan-2014 |
schwarze |
Always compare arch case-insensitively.
|
#
1.11 |
|
19-Jan-2014 |
schwarze |
Get rid of the local keys table, use the new mansearch_const.c. No functional change.
|
#
1.10 |
|
06-Jan-2014 |
schwarze |
Remove the redundant "file" column from the "mlinks" table. The contents can easily be reconstructed from sec, arch, name, form. Shrinks the database by another 3% in standard mode and 9% in -Q mode.
|
#
1.9 |
|
06-Jan-2014 |
schwarze |
Drop Nd from the mpages table, it is still in the keys table. This shrinks the database in standard mode by 3%, in -Q mode by 9%, without loss of functionality.
|
#
1.8 |
|
05-Jan-2014 |
schwarze |
Remove the obsolete file name column from the mpages table. This column wasn't helpful because one manpage can have multiple MLINKS. Use the file name column in the mlinks table, instead.
|
#
1.7 |
|
05-Jan-2014 |
schwarze |
Remove the obsolete sec and arch columns from the mpages table. They were confusing because a manpage can have MLINKS in different sections and architectures.
|
#
1.6 |
|
05-Jan-2014 |
schwarze |
Reimplement apropos -s NUM -S ARCH EXPR by internally converting it to apropos \( EXPR \) -a 'sec~^NUM$' -a 'arch~^(ARCH|any)$' in preparation for removal of sec and arch from the mpage table.
Almost no functional change except for the following bonus: This also makes sure that for cross-section and cross-arch MLINKs, all of the following work: apropos -s 1 encrypt apropos -s 8 encrypt apropos -s 1 makekey apropos -s 8 makekey Before this commit, they don't, neither for the old makewhatis(8)/apropos(1) combo nor for the new one.
While here, print error messages about invalid regexps to stderr.
|
#
1.5 |
|
05-Jan-2014 |
schwarze |
Put section and architecture info into the keys table, in preparation for removing them from the mpages table, aiming for cleaner and more uniform interfaces. Database growth is below 4%, part of which will be reclaimed.
As a bonus, this allows searches like: ./obj/apropos An=kettenis -a arch=ppc ./obj/apropos An=kettenis -a sec~[^4]
|
#
1.4 |
|
04-Jan-2014 |
schwarze |
New implementation of complex search criteria using \(, \), -a because the old implementation got lost in the Berkeley to SQLite switch. Note that this is not just feature creep, but required for upcoming database format cleanup and simplification.
The following takes less than half a second on my Z61m ThinkPad: time ./obj/apropos An=kettenis -a \( Cd=vnet -o Xr=vbus \)
|
#
1.3 |
|
31-Dec-2013 |
schwarze |
Experimental feature to let apropos(1) show different keys than .Nd. This really takes us beyond what grep -R /usr/*/man/ can do because now you can search for pages by *one* criterion and then display the contents of *another* macro from those pages, like in $ apropos -O Ox Fa~wchar to get an impression how long wide character handling is available.
|
#
1.2 |
|
31-Dec-2013 |
schwarze |
Split buildnames() out of mansearch(); the latter function is getting too long and unwieldy, but will grow more code soon. No functional change.
|
#
1.1 |
|
31-Dec-2013 |
schwarze |
Replace the Berkeley-DB based mandocdb(8) by an SQLite3-based version, aiming for more flexible development and optimization options. Kristaps started this during the summer 2012, i did some very heavy bugfixing during t2k13 and finally, during the last few days, got it to a state where it is ripe for in-tree development. Beware, neither the user interfaces nor the database formats are expected to be stable just yet.
Will not be installed or activated until further discussion. No functional change to mandoc(1).
"As long as it remains off until we decide the cost, fine." deraadt@
|
#
1.65 |
|
01-Jul-2019 |
schwarze |
delete trailing whitespace and space-tab sequences; no code change; patch from Michal Nowak <mnowak at startmail dot com> who found these with git pbchk in the illumos tree
|
#
1.64 |
|
30-Apr-2019 |
schwarze |
In man(1) mode, i.e. when asking for a single manual page by name, prefer file name matches over .Dt/.TH matches over first NAME matches over later NAME matches, but do not change the ordering for apropos(1) nor for man -a.
This reverts main.c rev. 1.213 and mansearch.h rev. 1.23 and includes a partial revert of mansearch.c rev. 1.62.
Regression reported by Lorenzo Beretta <loreb at github> as part of https://github.com/void-linux/void-packages/issues/9868 .
|
Revision tags: OPENBSD_6_5_BASE
|
#
1.63 |
|
13-Dec-2018 |
schwarze |
Cleanup, no functional change: Split the top level parser interface out of the utility header mandoc.h, into a new header mandoc_parse.h, for use in the main program and in the main parser only. Move enum mandoc_os into roff.h because struct roff_man is the place where it is stored. This allows removal of mandoc.h from seven files in low-level parsers and in formatters.
|
#
1.62 |
|
22-Nov-2018 |
schwarze |
In apropos(1) output, stop sorting .Nm search results by name priorities (bits). The obscure feature wasn't documented and merely confused people - for example Edward Tomasz Napierala <trasz at FreeBSD>, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227408.
Smaller patch provided by Yuri Pankov <yuripv at FreeBSD>, but i'm also retiring the now unused "bits" member from struct manpage. Simplification is good.
|
#
1.61 |
|
19-Nov-2018 |
schwarze |
Improve POSIX compliance by making case-insensitive extended regular expressions the default in man(1) -k searches, also matching what the man-db package used by many Linux distributions does.
Originally requested by Wolfram Schneider <wosch at FreeBSD> via Yuri Pankov <yuripv at FreeBSD>.
Feedback and OK cheloha@, and no objections when shown on tech@.
Thanks to cheloha@ for pointing out that POSIX requires this behaviour and for the suggestion to explicitly say that *extended* regular expressions are used here.
While here, unify spelling of case-[in]sensitive, fix a typo, update the EXAMPLES, and add a STANDARDS section.
|
Revision tags: OPENBSD_6_2_BASE OPENBSD_6_3_BASE OPENBSD_6_4_BASE
|
#
1.60 |
|
22-Aug-2017 |
schwarze |
When finding a bogus database entry, print a meaningful warning and skip the entry. Issue reported by espie@.
|
#
1.59 |
|
02-Aug-2017 |
schwarze |
No longer use names that only occur in the SYNOPSIS section as names for man(1) lookup. For OpenBSD base and Xenocara, that functionality was never intended to be required, and i just fixed the last handful of offenders using it - not counting the horribly ill-designed interfaces engine(3) and lh_new(3) which are impossible to properly document in the first place.
Of course, apropos(1) and whatis(1) continue to use SYNOPSIS .Nm, .Fn, and .Fo macros, so "man -k ENGINE_get_load_privkey_function" still works.
This change also gets rid of a few bogus warnings "cross reference to self" which actually are *not* to self, like in yp(8).
This former functionality was intended to help third-party software in the ports tree and on non-OpenBSD systems containing manual pages with incomplete or corrupt NAME sections. But it turned out it did more harm than good, and caused more confusion than relief, specifically for third party manuals and for maintainers of mandoc-portable on other operating systems. So kill it. Problems reported, among others, by Yuri Pankov (illumos).
OK jmc@
|
#
1.58 |
|
19-Jul-2017 |
schwarze |
Prefer arch-dependent over arch-independent pages if the name priority, the section number, and all names match. Changes little on installed systems except the ordering of apropos(1) results, because we install base and Xenocara manuals in different trees, but fixes lookup of pages like apm(4) vs. apm(4/amd64) in man.cgi(8). Issue discovered by martian67 on freenode and reported via tj@.
|
#
1.57 |
|
01-Jul-2017 |
schwarze |
Basic reporting of .Xrs to manual pages that don't exist in the base system, inspired by mdoclint(1).
We are able to do this because (1) the -mdoc parser, the -Tlint validator, and the man(1) manual page lookup code are all in the same program and (2) the mandoc.db(5) database format allows fast lookup.
Feedback from, previous versions tested by, and OK jmc@.
A few features will be added to this in the tree, step by step.
|
#
1.56 |
|
17-May-2017 |
schwarze |
Do not warn if a database file does not exist because that is simply equivalent to an empty database. Suggested by ajacoutot@
|
#
1.55 |
|
18-Apr-2017 |
schwarze |
The apropos(1) manual still documents the unary -i operator for regular expression search terms, but it appears that somewhere along the way, the implementation got lost, so restore it. Bug found while investigating other reports from Gonzalo Tornaria.
|
#
1.54 |
|
18-Apr-2017 |
schwarze |
Fix previous: i forgot explicit NUL termination; noticed by Gonzalo <Tornaria at cmat dot edu dot uy>, thanks!
|
#
1.53 |
|
17-Apr-2017 |
schwarze |
Restore -kO Nd, -kO Nm, -kO sec, and -kO arch to working order. They got broken in the SQLite removal. As opposed to the rest of -kO, they are no longer very useful, but they are certainly not supposed to fail assertions. Issue reported by Gonzalo Tornaria <tornaria at cmat dot edu dot uy>.
|
Revision tags: OPENBSD_6_1_BASE
|
#
1.52 |
|
03-Mar-2017 |
schwarze |
A missing initialization could randomly cause regular expression searches to be case-insensitive that ought to be case sensitive. Found by jsg@ with scan-build.
|
#
1.51 |
|
01-Aug-2016 |
schwarze |
Remove the dependency on SQLite without loss of functionality. Drop the obsolete names_check() now that we deleted MLINKS. Run "doas makewhatis" after compiling and installing this.
Earlier version tested by jmc@ and jturner@; "commit it all" deraadt@ "commit and dodge" krw@
|
Revision tags: OPENBSD_6_0_BASE
|
#
1.50 |
|
09-Jul-2016 |
schwarze |
getopt(3) is declared in <unistd.h>, and <getopt.h> is not needed; from Joerg Sonnenberger via Thomas Klausner, NetBSD.
|
Revision tags: OPENBSD_5_9_BASE
|
#
1.49 |
|
08-Jan-2016 |
schwarze |
Prefer warn(3) over perror(3) at the few places where it was used. It is useful to see the program name. Suggested by Christos Zoulas (NetBSD).
|
#
1.48 |
|
26-Nov-2015 |
schwarze |
No point in trying to go on when elementary database operations like preparing queries or binding variables fail; that won't yield useful results anyway but may generate huge pointless error messages. Issue reported by deraadt@.
|
#
1.47 |
|
13-Oct-2015 |
schwarze |
Reduce the amount of code by moving the three copies of the ohash callback functions into one common place, preparing for the use of ohash for some additional purposes. No functional change.
|
#
1.46 |
|
11-Oct-2015 |
schwarze |
Finally use __progname, err(3) and warn(3). That's more readable and less error-prone than fumbling around with argv[0], fprintf(3), strerror(3), perror(3), and exit(3). It also shortens the code by 50 lines.
It's a bad idea to boycott good interfaces merely because standards committees ignore them. Instead, it's the job of the portable distribution to provide compatibility modules for archaic systems (like commercial Solaris) that still don't have them. Actually, the compat code for the portable distribution already exists and will be committed right after this.
|
#
1.45 |
|
06-Oct-2015 |
schwarze |
modernize style: "return" is not a function; ok cmp(1)
|
Revision tags: OPENBSD_5_8_BASE
|
#
1.44 |
|
01-Apr-2015 |
schwarze |
Let man(1) and apropos(1) work even when the current directory is unusable: Only change back to the current directory when the directory was changed before and the next path is relative. This is now more similar to what makewhatis(8) does. Issue reported by espie@.
|
#
1.43 |
|
27-Mar-2015 |
schwarze |
Parse the new man.conf(5) "output" directive. The next step will be to actually use the parsed data.
|
#
1.42 |
|
11-Mar-2015 |
schwarze |
When interpreting the -O argument as a macro name fails, fall back to showing Nd rather than not showing anything. Issue reported by jmc@.
|
Revision tags: OPENBSD_5_7_BASE
|
#
1.41 |
|
27-Feb-2015 |
schwarze |
When man(1) and apropos(1) look for a file man1/foo.1 but it's unavailable, fall back to glob(man1/foo.*), which is more like what old man(1) did. Do this both for file names from the database and for fs_lookup(). This is relevant because some ports install files like man1/xset.1x. Regression reported by patrick keshishian <pkeshish at gmail dot com>.
|
#
1.40 |
|
20-Jan-2015 |
schwarze |
Make the man(1) and apropos(1) options -s and -S much less expensive: Do not append an SQL clause looking into the large "keys" table. Instead, filter the result of the SQL query in buildnames() where equivalent data from the much smaller "mlinks" table is already available for free.
This is relevant because man(1) uses the equivalent of "-S ${MACHINE}" by default since main.c rev. 1.122, to make sure that manuals for the current architecture are shown. With many ports installed, this patch can speed up man(1) by a factor of more than a hundred.
Slowness reported by Theo Buehler <theo at math dot ethz dot ch>, thanks!
|
#
1.39 |
|
06-Dec-2014 |
schwarze |
When opening mandoc.db fails, tell the user in which directory. Improving an unhelpful error message reported by millert@.
|
#
1.38 |
|
27-Nov-2014 |
schwarze |
Make makewhatis(8) understand .so links to .gz pages. Drop the FORM_GZ annotation in the mpages table; it is conceptually wrong because it ought to be in the mlinks table: An uncompressed .so link file can point to a compressed manual page file and vice versa. Besides, it is no longer needed because mparse_open() handles it all. Sprinkle some KNF while here.
|
#
1.37 |
|
18-Nov-2014 |
schwarze |
In man(1) mode, prefer file name matches over .Dt name matches over first .Nm entries over other NAME .Nm entries over SYNOPSIS .Nm entries. For example, this makes sure "man ypbind" does not return yp(8). Re-run "makewhatis" to profit from this change.
|
#
1.36 |
|
11-Nov-2014 |
schwarze |
In man(1) mode without -a, stop searching after the first manual tree that contained at least one match in order to not prefer mdoc(1) from ports over mdoc(7). As a bonus, this results in a speedup.
|
#
1.35 |
|
03-Sep-2014 |
schwarze |
If a manual page is installed gzip(1)ed, let makewhatis(8) take note in mandoc.db(5), such that man(1) -w and apropos(1) -w can report the correct filename. This is a prerequisite for letting apropos -a and man support gzip'ed manuals in the future, which doesn't work yet.
|
#
1.34 |
|
01-Sep-2014 |
schwarze |
In man(1) mode, change to the right directory before starting the parser, just like traditional man(1) does, such that .so links have a chance to work. After this point, we don't need the current directory for anything else before exit, so we don't need to worry about getting back and we can safely ignore failure. This lets man(1) find more Xenocara manuals, but not all of them yet. Other issues remain that need to be fixed, too.
|
#
1.33 |
|
27-Aug-2014 |
schwarze |
Add an implementation of man(1) into the /usr/bin/mandoc binary and provide a unified set of command line options for mandoc(1), man(1), apropos(1), and whatis(1), each option doing the same for all four. Not adding any completely new options, only extending exiting ones from one tool to the others. New options are: * apropos & whatis -acfkw (in the past, these were man(1) only) * apropos & whatis -a -IOTW (in the past, mandoc(1) only) * mandoc -ac (in the past, man(1) only) * man -IOTW (in the past, mandoc(1) only)
Before we can decide whether or not we want to replace src/usr.bin/man with this implementation, considerable bugfixing, testing, and performance measurements are needed, which i'd rather do in the tree than outside. Note that these bugs only affect the new man(1) mode, existing mandoc(1), apropos(1), and whatis(1) is fine.
The new functionality in mandoc(1), apropos(1), and whatis(1) is fully enabled. To play with the new man(1), you can try: # mv /usr/bin/man /usr/bin/oman # ln -s /usr/bin/mandoc /usr/bin/man
Positive feedback about the general direction from sthen@ and jmc@, and deraadt@ is not against it.
|
#
1.32 |
|
21-Aug-2014 |
schwarze |
Bugfix: make whatis(1) case-insensitive again. The traditional whatis(1) was case-insensitve and it's still documented that way, that but apparently got broken with or after the switch.
|
#
1.31 |
|
09-Aug-2014 |
schwarze |
mmap(2) requires MAP_PRIVATE ^ MAP_SHARED for flags; found by kristaps@ on Mac OS X
|
#
1.30 |
|
08-Aug-2014 |
schwarze |
Absurdly, the return value of sqlite3_column_text() is "const unsigned char *", which causes warnings with GCC on Linux. Explicitly cast to "const char *" to avoid this. Issue noticed by kristaps@.
|
Revision tags: OPENBSD_5_6_BASE
|
#
1.29 |
|
24-Jul-2014 |
schwarze |
Sort result pages first by section number, then by name. By moving the sort from cgi.c to mansearch.c, we get two advantages: Easier access to the data needed for sorting, in particular the section number, and the apropos(1) command line utility profits as well.
Feature requested by deraadt@.
|
#
1.28 |
|
12-Jul-2014 |
schwarze |
Fix whatis(1) to correctly match words instead of any substrings. While here, also provide an internal mode (MANSEARCH_MAN) to match complete names, to be used by man.cgi(8).
|
#
1.27 |
|
12-May-2014 |
espie |
adjust to ohash being in libutil now, and to the interface changes. fix potential integer overflows in memory allocation (mostly for pedagogical purposes, these are unlikely to overflow in practice) move the rest of lst.lib stuff into its own directory.
|
#
1.26 |
|
23-Apr-2014 |
schwarze |
Audit malloc(3)/calloc(3)/realloc(3) usage. * Change eight reallocs to reallocarray to be safe from overflows. * Change one malloc to reallocarray to be safe from overflows. * Change one calloc to reallocarray, no zeroing needed. * Change the order of arguments of three callocs (aesthetical).
|
#
1.25 |
|
23-Apr-2014 |
schwarze |
improve SQL style: avoid "SELECT *", be explicit in what columns we want; suggested by espie@.
|
#
1.24 |
|
20-Apr-2014 |
schwarze |
KNF: case (FOO): -> case FOO, remove /* LINTED */ and /* ARGSUSED */, remove trailing whitespace and blanks before tabs, improve some indenting; no functional change
|
#
1.23 |
|
17-Apr-2014 |
schwarze |
Garbage collect one pair of needless parentheses in SQL code generation; note this doesn't affect performance, SQLite generates the same byte code. While here, make the calls to exprspec() easier to understand.
|
#
1.22 |
|
16-Apr-2014 |
schwarze |
Rename the mpages.id column to mpages.pageid. There is no good reason to call this kid by a different name here than in all other tables. Easier to polish this now than after enabling.
|
#
1.21 |
|
16-Apr-2014 |
schwarze |
Pass the function flags SQLITE_UTF8 (because SQLITE_ANY is deprecated) and SQLITE_DETERMINISTIC when creating deterministic functions; best practice measure suggested by espie@ and jeremy@; as expected by jeremy@, no measurable effect on performance.
|
#
1.20 |
|
15-Apr-2014 |
schwarze |
At the end of mansearch(), fchdir() back to where we started from; this is cleaner and helps to not scatter gmon.out files all over the place when profiling.
|
#
1.19 |
|
11-Apr-2014 |
schwarze |
Further apropos(1) speed optimization was trickier than anticipated. Contrary to what i initially thought, almost all time is now spent inside sqlite3(3) routines, and i found no easy way calling less of them. However, sqlite(3) spends substantial time in malloc(3), and even more (twice that) in its immediate malloc wrapper, sqlite3MemMalloc(), keeping track of all individual malloc chunk sizes. Typically about 90% of the malloced memory is used for purposes of the pagecache.
By providing an mmap(3) MAP_ANON SQLITE_CONFIG_PAGECACHE, execution time decreases by 20-25% for simple (Nd and/or Nm) queries, 10-20% for non-NAME queries, and even apropos(1) resident memory size as reported by top(1) decreases by 20% for simple and by 60% for non-NAME queries. The new function, mansearch_setup(), spends no measurable time.
The pagesize chosen is optimal: * Substantially smaller pages yield no gain at all. * Larger pages provide no additional benefit and just waste memory.
The chosen number of pages in the cache is a compromise: * For simple queries, a handful of pages would suffice to get the full speed effect, at an apropos(1) resident memory size of about 2.0 MB. * For non-NAME queries, a large pagecache with 2k pages (2.5 MB) might gain a few more percent in speed, but at the expense of doubling the apropos(1) resident memory size for *all* queries. * The chosen number of 256 pages (330 kB) allows nearly full speed gain for all queries at the price of a 15% resident memory size increase.
|
#
1.18 |
|
10-Apr-2014 |
schwarze |
Next speed optimization step for the new apropos(1). Split manual names out of the common "keys" table into their own "names" table. This reduces standard apropos(1) search times (i.e. searching for names and descriptions only) by typically about 70% for the full /usr/share/man database. (Yes, that multiplies with the previous optimization step, so both together have reduced search times by a factor of more than six. I'm not done yet, expect more to come.)
Even with the minimal databases built with makewhatis(8) -Q, this step still reduces search times by 15-20%. For both cases, database sizes and build times hardly change (+/-2%).
|
#
1.17 |
|
09-Apr-2014 |
schwarze |
After careful gprof(1)ing of the new apropos(1), move the descriptions back from the keys table to the mpages table: I found a good way to still use them in searches, without complication of the code.
On my notebook, this reduces typical apropos(1) search times by about 40%, it reduces /usr/share/man database size by 6% in makewhatis(8) -Q mode and by 2% in standard mode (less overhead storing pointers to mpages), and it doesn't measurably change database build times (may even be going down by a percent or so because less data is being copied around in ohashes).
|
#
1.16 |
|
28-Mar-2014 |
schwarze |
Properly initialize malloc(3)ed memory. With this bug fix, partly unitialized memory could sometimes be returned, sometimes causing crashes by bogus free(3)s in apropos(1).
|
#
1.15 |
|
21-Mar-2014 |
schwarze |
avoid repetitive code for asprintf error handling
|
#
1.14 |
|
21-Mar-2014 |
schwarze |
The files mandoc.c and mandoc.h contained both specialised low-level functions used for multiple languages (mdoc, man, roff), for example mandoc_escape(), mandoc_getarg(), mandoc_eos(), and generic auxiliary functions. Split the auxiliaries out into their own file and header. While here, do some #include cleanup.
|
#
1.13 |
|
17-Mar-2014 |
schwarze |
in apropos(1) output, sort names and avoid multiple section numbers
|
Revision tags: OPENBSD_5_5_BASE
|
#
1.12 |
|
19-Jan-2014 |
schwarze |
Always compare arch case-insensitively.
|
#
1.11 |
|
19-Jan-2014 |
schwarze |
Get rid of the local keys table, use the new mansearch_const.c. No functional change.
|
#
1.10 |
|
06-Jan-2014 |
schwarze |
Remove the redundant "file" column from the "mlinks" table. The contents can easily be reconstructed from sec, arch, name, form. Shrinks the database by another 3% in standard mode and 9% in -Q mode.
|
#
1.9 |
|
06-Jan-2014 |
schwarze |
Drop Nd from the mpages table, it is still in the keys table. This shrinks the database in standard mode by 3%, in -Q mode by 9%, without loss of functionality.
|
#
1.8 |
|
05-Jan-2014 |
schwarze |
Remove the obsolete file name column from the mpages table. This column wasn't helpful because one manpage can have multiple MLINKS. Use the file name column in the mlinks table, instead.
|
#
1.7 |
|
05-Jan-2014 |
schwarze |
Remove the obsolete sec and arch columns from the mpages table. They were confusing because a manpage can have MLINKS in different sections and architectures.
|
#
1.6 |
|
05-Jan-2014 |
schwarze |
Reimplement apropos -s NUM -S ARCH EXPR by internally converting it to apropos \( EXPR \) -a 'sec~^NUM$' -a 'arch~^(ARCH|any)$' in preparation for removal of sec and arch from the mpage table.
Almost no functional change except for the following bonus: This also makes sure that for cross-section and cross-arch MLINKs, all of the following work: apropos -s 1 encrypt apropos -s 8 encrypt apropos -s 1 makekey apropos -s 8 makekey Before this commit, they don't, neither for the old makewhatis(8)/apropos(1) combo nor for the new one.
While here, print error messages about invalid regexps to stderr.
|
#
1.5 |
|
05-Jan-2014 |
schwarze |
Put section and architecture info into the keys table, in preparation for removing them from the mpages table, aiming for cleaner and more uniform interfaces. Database growth is below 4%, part of which will be reclaimed.
As a bonus, this allows searches like: ./obj/apropos An=kettenis -a arch=ppc ./obj/apropos An=kettenis -a sec~[^4]
|
#
1.4 |
|
04-Jan-2014 |
schwarze |
New implementation of complex search criteria using \(, \), -a because the old implementation got lost in the Berkeley to SQLite switch. Note that this is not just feature creep, but required for upcoming database format cleanup and simplification.
The following takes less than half a second on my Z61m ThinkPad: time ./obj/apropos An=kettenis -a \( Cd=vnet -o Xr=vbus \)
|
#
1.3 |
|
31-Dec-2013 |
schwarze |
Experimental feature to let apropos(1) show different keys than .Nd. This really takes us beyond what grep -R /usr/*/man/ can do because now you can search for pages by *one* criterion and then display the contents of *another* macro from those pages, like in $ apropos -O Ox Fa~wchar to get an impression how long wide character handling is available.
|
#
1.2 |
|
31-Dec-2013 |
schwarze |
Split buildnames() out of mansearch(); the latter function is getting too long and unwieldy, but will grow more code soon. No functional change.
|
#
1.1 |
|
31-Dec-2013 |
schwarze |
Replace the Berkeley-DB based mandocdb(8) by an SQLite3-based version, aiming for more flexible development and optimization options. Kristaps started this during the summer 2012, i did some very heavy bugfixing during t2k13 and finally, during the last few days, got it to a state where it is ripe for in-tree development. Beware, neither the user interfaces nor the database formats are expected to be stable just yet.
Will not be installed or activated until further discussion. No functional change to mandoc(1).
"As long as it remains off until we decide the cost, fine." deraadt@
|
#
1.64 |
|
30-Apr-2019 |
schwarze |
In man(1) mode, i.e. when asking for a single manual page by name, prefer file name matches over .Dt/.TH matches over first NAME matches over later NAME matches, but do not change the ordering for apropos(1) nor for man -a.
This reverts main.c rev. 1.213 and mansearch.h rev. 1.23 and includes a partial revert of mansearch.c rev. 1.62.
Regression reported by Lorenzo Beretta <loreb at github> as part of https://github.com/void-linux/void-packages/issues/9868 .
|
Revision tags: OPENBSD_6_5_BASE
|
#
1.63 |
|
13-Dec-2018 |
schwarze |
Cleanup, no functional change: Split the top level parser interface out of the utility header mandoc.h, into a new header mandoc_parse.h, for use in the main program and in the main parser only. Move enum mandoc_os into roff.h because struct roff_man is the place where it is stored. This allows removal of mandoc.h from seven files in low-level parsers and in formatters.
|
#
1.62 |
|
22-Nov-2018 |
schwarze |
In apropos(1) output, stop sorting .Nm search results by name priorities (bits). The obscure feature wasn't documented and merely confused people - for example Edward Tomasz Napierala <trasz at FreeBSD>, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227408.
Smaller patch provided by Yuri Pankov <yuripv at FreeBSD>, but i'm also retiring the now unused "bits" member from struct manpage. Simplification is good.
|
#
1.61 |
|
19-Nov-2018 |
schwarze |
Improve POSIX compliance by making case-insensitive extended regular expressions the default in man(1) -k searches, also matching what the man-db package used by many Linux distributions does.
Originally requested by Wolfram Schneider <wosch at FreeBSD> via Yuri Pankov <yuripv at FreeBSD>.
Feedback and OK cheloha@, and no objections when shown on tech@.
Thanks to cheloha@ for pointing out that POSIX requires this behaviour and for the suggestion to explicitly say that *extended* regular expressions are used here.
While here, unify spelling of case-[in]sensitive, fix a typo, update the EXAMPLES, and add a STANDARDS section.
|
Revision tags: OPENBSD_6_2_BASE OPENBSD_6_3_BASE OPENBSD_6_4_BASE
|
#
1.60 |
|
22-Aug-2017 |
schwarze |
When finding a bogus database entry, print a meaningful warning and skip the entry. Issue reported by espie@.
|
#
1.59 |
|
02-Aug-2017 |
schwarze |
No longer use names that only occur in the SYNOPSIS section as names for man(1) lookup. For OpenBSD base and Xenocara, that functionality was never intended to be required, and i just fixed the last handful of offenders using it - not counting the horribly ill-designed interfaces engine(3) and lh_new(3) which are impossible to properly document in the first place.
Of course, apropos(1) and whatis(1) continue to use SYNOPSIS .Nm, .Fn, and .Fo macros, so "man -k ENGINE_get_load_privkey_function" still works.
This change also gets rid of a few bogus warnings "cross reference to self" which actually are *not* to self, like in yp(8).
This former functionality was intended to help third-party software in the ports tree and on non-OpenBSD systems containing manual pages with incomplete or corrupt NAME sections. But it turned out it did more harm than good, and caused more confusion than relief, specifically for third party manuals and for maintainers of mandoc-portable on other operating systems. So kill it. Problems reported, among others, by Yuri Pankov (illumos).
OK jmc@
|
#
1.58 |
|
19-Jul-2017 |
schwarze |
Prefer arch-dependent over arch-independent pages if the name priority, the section number, and all names match. Changes little on installed systems except the ordering of apropos(1) results, because we install base and Xenocara manuals in different trees, but fixes lookup of pages like apm(4) vs. apm(4/amd64) in man.cgi(8). Issue discovered by martian67 on freenode and reported via tj@.
|
#
1.57 |
|
01-Jul-2017 |
schwarze |
Basic reporting of .Xrs to manual pages that don't exist in the base system, inspired by mdoclint(1).
We are able to do this because (1) the -mdoc parser, the -Tlint validator, and the man(1) manual page lookup code are all in the same program and (2) the mandoc.db(5) database format allows fast lookup.
Feedback from, previous versions tested by, and OK jmc@.
A few features will be added to this in the tree, step by step.
|
#
1.56 |
|
17-May-2017 |
schwarze |
Do not warn if a database file does not exist because that is simply equivalent to an empty database. Suggested by ajacoutot@
|
#
1.55 |
|
18-Apr-2017 |
schwarze |
The apropos(1) manual still documents the unary -i operator for regular expression search terms, but it appears that somewhere along the way, the implementation got lost, so restore it. Bug found while investigating other reports from Gonzalo Tornaria.
|
#
1.54 |
|
18-Apr-2017 |
schwarze |
Fix previous: i forgot explicit NUL termination; noticed by Gonzalo <Tornaria at cmat dot edu dot uy>, thanks!
|
#
1.53 |
|
17-Apr-2017 |
schwarze |
Restore -kO Nd, -kO Nm, -kO sec, and -kO arch to working order. They got broken in the SQLite removal. As opposed to the rest of -kO, they are no longer very useful, but they are certainly not supposed to fail assertions. Issue reported by Gonzalo Tornaria <tornaria at cmat dot edu dot uy>.
|
Revision tags: OPENBSD_6_1_BASE
|
#
1.52 |
|
03-Mar-2017 |
schwarze |
A missing initialization could randomly cause regular expression searches to be case-insensitive that ought to be case sensitive. Found by jsg@ with scan-build.
|
#
1.51 |
|
01-Aug-2016 |
schwarze |
Remove the dependency on SQLite without loss of functionality. Drop the obsolete names_check() now that we deleted MLINKS. Run "doas makewhatis" after compiling and installing this.
Earlier version tested by jmc@ and jturner@; "commit it all" deraadt@ "commit and dodge" krw@
|
Revision tags: OPENBSD_6_0_BASE
|
#
1.50 |
|
09-Jul-2016 |
schwarze |
getopt(3) is declared in <unistd.h>, and <getopt.h> is not needed; from Joerg Sonnenberger via Thomas Klausner, NetBSD.
|
Revision tags: OPENBSD_5_9_BASE
|
#
1.49 |
|
08-Jan-2016 |
schwarze |
Prefer warn(3) over perror(3) at the few places where it was used. It is useful to see the program name. Suggested by Christos Zoulas (NetBSD).
|
#
1.48 |
|
26-Nov-2015 |
schwarze |
No point in trying to go on when elementary database operations like preparing queries or binding variables fail; that won't yield useful results anyway but may generate huge pointless error messages. Issue reported by deraadt@.
|
#
1.47 |
|
13-Oct-2015 |
schwarze |
Reduce the amount of code by moving the three copies of the ohash callback functions into one common place, preparing for the use of ohash for some additional purposes. No functional change.
|
#
1.46 |
|
11-Oct-2015 |
schwarze |
Finally use __progname, err(3) and warn(3). That's more readable and less error-prone than fumbling around with argv[0], fprintf(3), strerror(3), perror(3), and exit(3). It also shortens the code by 50 lines.
It's a bad idea to boycott good interfaces merely because standards committees ignore them. Instead, it's the job of the portable distribution to provide compatibility modules for archaic systems (like commercial Solaris) that still don't have them. Actually, the compat code for the portable distribution already exists and will be committed right after this.
|
#
1.45 |
|
06-Oct-2015 |
schwarze |
modernize style: "return" is not a function; ok cmp(1)
|
Revision tags: OPENBSD_5_8_BASE
|
#
1.44 |
|
01-Apr-2015 |
schwarze |
Let man(1) and apropos(1) work even when the current directory is unusable: Only change back to the current directory when the directory was changed before and the next path is relative. This is now more similar to what makewhatis(8) does. Issue reported by espie@.
|
#
1.43 |
|
27-Mar-2015 |
schwarze |
Parse the new man.conf(5) "output" directive. The next step will be to actually use the parsed data.
|
#
1.42 |
|
11-Mar-2015 |
schwarze |
When interpreting the -O argument as a macro name fails, fall back to showing Nd rather than not showing anything. Issue reported by jmc@.
|
Revision tags: OPENBSD_5_7_BASE
|
#
1.41 |
|
27-Feb-2015 |
schwarze |
When man(1) and apropos(1) look for a file man1/foo.1 but it's unavailable, fall back to glob(man1/foo.*), which is more like what old man(1) did. Do this both for file names from the database and for fs_lookup(). This is relevant because some ports install files like man1/xset.1x. Regression reported by patrick keshishian <pkeshish at gmail dot com>.
|
#
1.40 |
|
20-Jan-2015 |
schwarze |
Make the man(1) and apropos(1) options -s and -S much less expensive: Do not append an SQL clause looking into the large "keys" table. Instead, filter the result of the SQL query in buildnames() where equivalent data from the much smaller "mlinks" table is already available for free.
This is relevant because man(1) uses the equivalent of "-S ${MACHINE}" by default since main.c rev. 1.122, to make sure that manuals for the current architecture are shown. With many ports installed, this patch can speed up man(1) by a factor of more than a hundred.
Slowness reported by Theo Buehler <theo at math dot ethz dot ch>, thanks!
|
#
1.39 |
|
06-Dec-2014 |
schwarze |
When opening mandoc.db fails, tell the user in which directory. Improving an unhelpful error message reported by millert@.
|
#
1.38 |
|
27-Nov-2014 |
schwarze |
Make makewhatis(8) understand .so links to .gz pages. Drop the FORM_GZ annotation in the mpages table; it is conceptually wrong because it ought to be in the mlinks table: An uncompressed .so link file can point to a compressed manual page file and vice versa. Besides, it is no longer needed because mparse_open() handles it all. Sprinkle some KNF while here.
|
#
1.37 |
|
18-Nov-2014 |
schwarze |
In man(1) mode, prefer file name matches over .Dt name matches over first .Nm entries over other NAME .Nm entries over SYNOPSIS .Nm entries. For example, this makes sure "man ypbind" does not return yp(8). Re-run "makewhatis" to profit from this change.
|
#
1.36 |
|
11-Nov-2014 |
schwarze |
In man(1) mode without -a, stop searching after the first manual tree that contained at least one match in order to not prefer mdoc(1) from ports over mdoc(7). As a bonus, this results in a speedup.
|
#
1.35 |
|
03-Sep-2014 |
schwarze |
If a manual page is installed gzip(1)ed, let makewhatis(8) take note in mandoc.db(5), such that man(1) -w and apropos(1) -w can report the correct filename. This is a prerequisite for letting apropos -a and man support gzip'ed manuals in the future, which doesn't work yet.
|
#
1.34 |
|
01-Sep-2014 |
schwarze |
In man(1) mode, change to the right directory before starting the parser, just like traditional man(1) does, such that .so links have a chance to work. After this point, we don't need the current directory for anything else before exit, so we don't need to worry about getting back and we can safely ignore failure. This lets man(1) find more Xenocara manuals, but not all of them yet. Other issues remain that need to be fixed, too.
|
#
1.33 |
|
27-Aug-2014 |
schwarze |
Add an implementation of man(1) into the /usr/bin/mandoc binary and provide a unified set of command line options for mandoc(1), man(1), apropos(1), and whatis(1), each option doing the same for all four. Not adding any completely new options, only extending exiting ones from one tool to the others. New options are: * apropos & whatis -acfkw (in the past, these were man(1) only) * apropos & whatis -a -IOTW (in the past, mandoc(1) only) * mandoc -ac (in the past, man(1) only) * man -IOTW (in the past, mandoc(1) only)
Before we can decide whether or not we want to replace src/usr.bin/man with this implementation, considerable bugfixing, testing, and performance measurements are needed, which i'd rather do in the tree than outside. Note that these bugs only affect the new man(1) mode, existing mandoc(1), apropos(1), and whatis(1) is fine.
The new functionality in mandoc(1), apropos(1), and whatis(1) is fully enabled. To play with the new man(1), you can try: # mv /usr/bin/man /usr/bin/oman # ln -s /usr/bin/mandoc /usr/bin/man
Positive feedback about the general direction from sthen@ and jmc@, and deraadt@ is not against it.
|
#
1.32 |
|
21-Aug-2014 |
schwarze |
Bugfix: make whatis(1) case-insensitive again. The traditional whatis(1) was case-insensitve and it's still documented that way, that but apparently got broken with or after the switch.
|
#
1.31 |
|
09-Aug-2014 |
schwarze |
mmap(2) requires MAP_PRIVATE ^ MAP_SHARED for flags; found by kristaps@ on Mac OS X
|
#
1.30 |
|
08-Aug-2014 |
schwarze |
Absurdly, the return value of sqlite3_column_text() is "const unsigned char *", which causes warnings with GCC on Linux. Explicitly cast to "const char *" to avoid this. Issue noticed by kristaps@.
|
Revision tags: OPENBSD_5_6_BASE
|
#
1.29 |
|
24-Jul-2014 |
schwarze |
Sort result pages first by section number, then by name. By moving the sort from cgi.c to mansearch.c, we get two advantages: Easier access to the data needed for sorting, in particular the section number, and the apropos(1) command line utility profits as well.
Feature requested by deraadt@.
|
#
1.28 |
|
12-Jul-2014 |
schwarze |
Fix whatis(1) to correctly match words instead of any substrings. While here, also provide an internal mode (MANSEARCH_MAN) to match complete names, to be used by man.cgi(8).
|
#
1.27 |
|
12-May-2014 |
espie |
adjust to ohash being in libutil now, and to the interface changes. fix potential integer overflows in memory allocation (mostly for pedagogical purposes, these are unlikely to overflow in practice) move the rest of lst.lib stuff into its own directory.
|
#
1.26 |
|
23-Apr-2014 |
schwarze |
Audit malloc(3)/calloc(3)/realloc(3) usage. * Change eight reallocs to reallocarray to be safe from overflows. * Change one malloc to reallocarray to be safe from overflows. * Change one calloc to reallocarray, no zeroing needed. * Change the order of arguments of three callocs (aesthetical).
|
#
1.25 |
|
23-Apr-2014 |
schwarze |
improve SQL style: avoid "SELECT *", be explicit in what columns we want; suggested by espie@.
|
#
1.24 |
|
20-Apr-2014 |
schwarze |
KNF: case (FOO): -> case FOO, remove /* LINTED */ and /* ARGSUSED */, remove trailing whitespace and blanks before tabs, improve some indenting; no functional change
|
#
1.23 |
|
17-Apr-2014 |
schwarze |
Garbage collect one pair of needless parentheses in SQL code generation; note this doesn't affect performance, SQLite generates the same byte code. While here, make the calls to exprspec() easier to understand.
|
#
1.22 |
|
16-Apr-2014 |
schwarze |
Rename the mpages.id column to mpages.pageid. There is no good reason to call this kid by a different name here than in all other tables. Easier to polish this now than after enabling.
|
#
1.21 |
|
16-Apr-2014 |
schwarze |
Pass the function flags SQLITE_UTF8 (because SQLITE_ANY is deprecated) and SQLITE_DETERMINISTIC when creating deterministic functions; best practice measure suggested by espie@ and jeremy@; as expected by jeremy@, no measurable effect on performance.
|
#
1.20 |
|
15-Apr-2014 |
schwarze |
At the end of mansearch(), fchdir() back to where we started from; this is cleaner and helps to not scatter gmon.out files all over the place when profiling.
|
#
1.19 |
|
11-Apr-2014 |
schwarze |
Further apropos(1) speed optimization was trickier than anticipated. Contrary to what i initially thought, almost all time is now spent inside sqlite3(3) routines, and i found no easy way calling less of them. However, sqlite(3) spends substantial time in malloc(3), and even more (twice that) in its immediate malloc wrapper, sqlite3MemMalloc(), keeping track of all individual malloc chunk sizes. Typically about 90% of the malloced memory is used for purposes of the pagecache.
By providing an mmap(3) MAP_ANON SQLITE_CONFIG_PAGECACHE, execution time decreases by 20-25% for simple (Nd and/or Nm) queries, 10-20% for non-NAME queries, and even apropos(1) resident memory size as reported by top(1) decreases by 20% for simple and by 60% for non-NAME queries. The new function, mansearch_setup(), spends no measurable time.
The pagesize chosen is optimal: * Substantially smaller pages yield no gain at all. * Larger pages provide no additional benefit and just waste memory.
The chosen number of pages in the cache is a compromise: * For simple queries, a handful of pages would suffice to get the full speed effect, at an apropos(1) resident memory size of about 2.0 MB. * For non-NAME queries, a large pagecache with 2k pages (2.5 MB) might gain a few more percent in speed, but at the expense of doubling the apropos(1) resident memory size for *all* queries. * The chosen number of 256 pages (330 kB) allows nearly full speed gain for all queries at the price of a 15% resident memory size increase.
|
#
1.18 |
|
10-Apr-2014 |
schwarze |
Next speed optimization step for the new apropos(1). Split manual names out of the common "keys" table into their own "names" table. This reduces standard apropos(1) search times (i.e. searching for names and descriptions only) by typically about 70% for the full /usr/share/man database. (Yes, that multiplies with the previous optimization step, so both together have reduced search times by a factor of more than six. I'm not done yet, expect more to come.)
Even with the minimal databases built with makewhatis(8) -Q, this step still reduces search times by 15-20%. For both cases, database sizes and build times hardly change (+/-2%).
|
#
1.17 |
|
09-Apr-2014 |
schwarze |
After careful gprof(1)ing of the new apropos(1), move the descriptions back from the keys table to the mpages table: I found a good way to still use them in searches, without complication of the code.
On my notebook, this reduces typical apropos(1) search times by about 40%, it reduces /usr/share/man database size by 6% in makewhatis(8) -Q mode and by 2% in standard mode (less overhead storing pointers to mpages), and it doesn't measurably change database build times (may even be going down by a percent or so because less data is being copied around in ohashes).
|
#
1.16 |
|
28-Mar-2014 |
schwarze |
Properly initialize malloc(3)ed memory. With this bug fix, partly unitialized memory could sometimes be returned, sometimes causing crashes by bogus free(3)s in apropos(1).
|
#
1.15 |
|
21-Mar-2014 |
schwarze |
avoid repetitive code for asprintf error handling
|
#
1.14 |
|
21-Mar-2014 |
schwarze |
The files mandoc.c and mandoc.h contained both specialised low-level functions used for multiple languages (mdoc, man, roff), for example mandoc_escape(), mandoc_getarg(), mandoc_eos(), and generic auxiliary functions. Split the auxiliaries out into their own file and header. While here, do some #include cleanup.
|
#
1.13 |
|
17-Mar-2014 |
schwarze |
in apropos(1) output, sort names and avoid multiple section numbers
|
Revision tags: OPENBSD_5_5_BASE
|
#
1.12 |
|
19-Jan-2014 |
schwarze |
Always compare arch case-insensitively.
|
#
1.11 |
|
19-Jan-2014 |
schwarze |
Get rid of the local keys table, use the new mansearch_const.c. No functional change.
|
#
1.10 |
|
06-Jan-2014 |
schwarze |
Remove the redundant "file" column from the "mlinks" table. The contents can easily be reconstructed from sec, arch, name, form. Shrinks the database by another 3% in standard mode and 9% in -Q mode.
|
#
1.9 |
|
06-Jan-2014 |
schwarze |
Drop Nd from the mpages table, it is still in the keys table. This shrinks the database in standard mode by 3%, in -Q mode by 9%, without loss of functionality.
|
#
1.8 |
|
05-Jan-2014 |
schwarze |
Remove the obsolete file name column from the mpages table. This column wasn't helpful because one manpage can have multiple MLINKS. Use the file name column in the mlinks table, instead.
|
#
1.7 |
|
05-Jan-2014 |
schwarze |
Remove the obsolete sec and arch columns from the mpages table. They were confusing because a manpage can have MLINKS in different sections and architectures.
|
#
1.6 |
|
05-Jan-2014 |
schwarze |
Reimplement apropos -s NUM -S ARCH EXPR by internally converting it to apropos \( EXPR \) -a 'sec~^NUM$' -a 'arch~^(ARCH|any)$' in preparation for removal of sec and arch from the mpage table.
Almost no functional change except for the following bonus: This also makes sure that for cross-section and cross-arch MLINKs, all of the following work: apropos -s 1 encrypt apropos -s 8 encrypt apropos -s 1 makekey apropos -s 8 makekey Before this commit, they don't, neither for the old makewhatis(8)/apropos(1) combo nor for the new one.
While here, print error messages about invalid regexps to stderr.
|
#
1.5 |
|
05-Jan-2014 |
schwarze |
Put section and architecture info into the keys table, in preparation for removing them from the mpages table, aiming for cleaner and more uniform interfaces. Database growth is below 4%, part of which will be reclaimed.
As a bonus, this allows searches like: ./obj/apropos An=kettenis -a arch=ppc ./obj/apropos An=kettenis -a sec~[^4]
|
#
1.4 |
|
04-Jan-2014 |
schwarze |
New implementation of complex search criteria using \(, \), -a because the old implementation got lost in the Berkeley to SQLite switch. Note that this is not just feature creep, but required for upcoming database format cleanup and simplification.
The following takes less than half a second on my Z61m ThinkPad: time ./obj/apropos An=kettenis -a \( Cd=vnet -o Xr=vbus \)
|
#
1.3 |
|
31-Dec-2013 |
schwarze |
Experimental feature to let apropos(1) show different keys than .Nd. This really takes us beyond what grep -R /usr/*/man/ can do because now you can search for pages by *one* criterion and then display the contents of *another* macro from those pages, like in $ apropos -O Ox Fa~wchar to get an impression how long wide character handling is available.
|
#
1.2 |
|
31-Dec-2013 |
schwarze |
Split buildnames() out of mansearch(); the latter function is getting too long and unwieldy, but will grow more code soon. No functional change.
|
#
1.1 |
|
31-Dec-2013 |
schwarze |
Replace the Berkeley-DB based mandocdb(8) by an SQLite3-based version, aiming for more flexible development and optimization options. Kristaps started this during the summer 2012, i did some very heavy bugfixing during t2k13 and finally, during the last few days, got it to a state where it is ripe for in-tree development. Beware, neither the user interfaces nor the database formats are expected to be stable just yet.
Will not be installed or activated until further discussion. No functional change to mandoc(1).
"As long as it remains off until we decide the cost, fine." deraadt@
|
#
1.63 |
|
13-Dec-2018 |
schwarze |
Cleanup, no functional change: Split the top level parser interface out of the utility header mandoc.h, into a new header mandoc_parse.h, for use in the main program and in the main parser only. Move enum mandoc_os into roff.h because struct roff_man is the place where it is stored. This allows removal of mandoc.h from seven files in low-level parsers and in formatters.
|
#
1.62 |
|
22-Nov-2018 |
schwarze |
In apropos(1) output, stop sorting .Nm search results by name priorities (bits). The obscure feature wasn't documented and merely confused people - for example Edward Tomasz Napierala <trasz at FreeBSD>, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227408.
Smaller patch provided by Yuri Pankov <yuripv at FreeBSD>, but i'm also retiring the now unused "bits" member from struct manpage. Simplification is good.
|
#
1.61 |
|
19-Nov-2018 |
schwarze |
Improve POSIX compliance by making case-insensitive extended regular expressions the default in man(1) -k searches, also matching what the man-db package used by many Linux distributions does.
Originally requested by Wolfram Schneider <wosch at FreeBSD> via Yuri Pankov <yuripv at FreeBSD>.
Feedback and OK cheloha@, and no objections when shown on tech@.
Thanks to cheloha@ for pointing out that POSIX requires this behaviour and for the suggestion to explicitly say that *extended* regular expressions are used here.
While here, unify spelling of case-[in]sensitive, fix a typo, update the EXAMPLES, and add a STANDARDS section.
|
Revision tags: OPENBSD_6_2_BASE OPENBSD_6_3_BASE OPENBSD_6_4_BASE
|
#
1.60 |
|
22-Aug-2017 |
schwarze |
When finding a bogus database entry, print a meaningful warning and skip the entry. Issue reported by espie@.
|
#
1.59 |
|
02-Aug-2017 |
schwarze |
No longer use names that only occur in the SYNOPSIS section as names for man(1) lookup. For OpenBSD base and Xenocara, that functionality was never intended to be required, and i just fixed the last handful of offenders using it - not counting the horribly ill-designed interfaces engine(3) and lh_new(3) which are impossible to properly document in the first place.
Of course, apropos(1) and whatis(1) continue to use SYNOPSIS .Nm, .Fn, and .Fo macros, so "man -k ENGINE_get_load_privkey_function" still works.
This change also gets rid of a few bogus warnings "cross reference to self" which actually are *not* to self, like in yp(8).
This former functionality was intended to help third-party software in the ports tree and on non-OpenBSD systems containing manual pages with incomplete or corrupt NAME sections. But it turned out it did more harm than good, and caused more confusion than relief, specifically for third party manuals and for maintainers of mandoc-portable on other operating systems. So kill it. Problems reported, among others, by Yuri Pankov (illumos).
OK jmc@
|
#
1.58 |
|
19-Jul-2017 |
schwarze |
Prefer arch-dependent over arch-independent pages if the name priority, the section number, and all names match. Changes little on installed systems except the ordering of apropos(1) results, because we install base and Xenocara manuals in different trees, but fixes lookup of pages like apm(4) vs. apm(4/amd64) in man.cgi(8). Issue discovered by martian67 on freenode and reported via tj@.
|
#
1.57 |
|
01-Jul-2017 |
schwarze |
Basic reporting of .Xrs to manual pages that don't exist in the base system, inspired by mdoclint(1).
We are able to do this because (1) the -mdoc parser, the -Tlint validator, and the man(1) manual page lookup code are all in the same program and (2) the mandoc.db(5) database format allows fast lookup.
Feedback from, previous versions tested by, and OK jmc@.
A few features will be added to this in the tree, step by step.
|
#
1.56 |
|
17-May-2017 |
schwarze |
Do not warn if a database file does not exist because that is simply equivalent to an empty database. Suggested by ajacoutot@
|
#
1.55 |
|
18-Apr-2017 |
schwarze |
The apropos(1) manual still documents the unary -i operator for regular expression search terms, but it appears that somewhere along the way, the implementation got lost, so restore it. Bug found while investigating other reports from Gonzalo Tornaria.
|
#
1.54 |
|
18-Apr-2017 |
schwarze |
Fix previous: i forgot explicit NUL termination; noticed by Gonzalo <Tornaria at cmat dot edu dot uy>, thanks!
|
#
1.53 |
|
17-Apr-2017 |
schwarze |
Restore -kO Nd, -kO Nm, -kO sec, and -kO arch to working order. They got broken in the SQLite removal. As opposed to the rest of -kO, they are no longer very useful, but they are certainly not supposed to fail assertions. Issue reported by Gonzalo Tornaria <tornaria at cmat dot edu dot uy>.
|
Revision tags: OPENBSD_6_1_BASE
|
#
1.52 |
|
03-Mar-2017 |
schwarze |
A missing initialization could randomly cause regular expression searches to be case-insensitive that ought to be case sensitive. Found by jsg@ with scan-build.
|
#
1.51 |
|
01-Aug-2016 |
schwarze |
Remove the dependency on SQLite without loss of functionality. Drop the obsolete names_check() now that we deleted MLINKS. Run "doas makewhatis" after compiling and installing this.
Earlier version tested by jmc@ and jturner@; "commit it all" deraadt@ "commit and dodge" krw@
|
Revision tags: OPENBSD_6_0_BASE
|
#
1.50 |
|
09-Jul-2016 |
schwarze |
getopt(3) is declared in <unistd.h>, and <getopt.h> is not needed; from Joerg Sonnenberger via Thomas Klausner, NetBSD.
|
Revision tags: OPENBSD_5_9_BASE
|
#
1.49 |
|
08-Jan-2016 |
schwarze |
Prefer warn(3) over perror(3) at the few places where it was used. It is useful to see the program name. Suggested by Christos Zoulas (NetBSD).
|
#
1.48 |
|
26-Nov-2015 |
schwarze |
No point in trying to go on when elementary database operations like preparing queries or binding variables fail; that won't yield useful results anyway but may generate huge pointless error messages. Issue reported by deraadt@.
|
#
1.47 |
|
13-Oct-2015 |
schwarze |
Reduce the amount of code by moving the three copies of the ohash callback functions into one common place, preparing for the use of ohash for some additional purposes. No functional change.
|
#
1.46 |
|
11-Oct-2015 |
schwarze |
Finally use __progname, err(3) and warn(3). That's more readable and less error-prone than fumbling around with argv[0], fprintf(3), strerror(3), perror(3), and exit(3). It also shortens the code by 50 lines.
It's a bad idea to boycott good interfaces merely because standards committees ignore them. Instead, it's the job of the portable distribution to provide compatibility modules for archaic systems (like commercial Solaris) that still don't have them. Actually, the compat code for the portable distribution already exists and will be committed right after this.
|
#
1.45 |
|
06-Oct-2015 |
schwarze |
modernize style: "return" is not a function; ok cmp(1)
|
Revision tags: OPENBSD_5_8_BASE
|
#
1.44 |
|
01-Apr-2015 |
schwarze |
Let man(1) and apropos(1) work even when the current directory is unusable: Only change back to the current directory when the directory was changed before and the next path is relative. This is now more similar to what makewhatis(8) does. Issue reported by espie@.
|
#
1.43 |
|
27-Mar-2015 |
schwarze |
Parse the new man.conf(5) "output" directive. The next step will be to actually use the parsed data.
|
#
1.42 |
|
11-Mar-2015 |
schwarze |
When interpreting the -O argument as a macro name fails, fall back to showing Nd rather than not showing anything. Issue reported by jmc@.
|
Revision tags: OPENBSD_5_7_BASE
|
#
1.41 |
|
27-Feb-2015 |
schwarze |
When man(1) and apropos(1) look for a file man1/foo.1 but it's unavailable, fall back to glob(man1/foo.*), which is more like what old man(1) did. Do this both for file names from the database and for fs_lookup(). This is relevant because some ports install files like man1/xset.1x. Regression reported by patrick keshishian <pkeshish at gmail dot com>.
|
#
1.40 |
|
20-Jan-2015 |
schwarze |
Make the man(1) and apropos(1) options -s and -S much less expensive: Do not append an SQL clause looking into the large "keys" table. Instead, filter the result of the SQL query in buildnames() where equivalent data from the much smaller "mlinks" table is already available for free.
This is relevant because man(1) uses the equivalent of "-S ${MACHINE}" by default since main.c rev. 1.122, to make sure that manuals for the current architecture are shown. With many ports installed, this patch can speed up man(1) by a factor of more than a hundred.
Slowness reported by Theo Buehler <theo at math dot ethz dot ch>, thanks!
|
#
1.39 |
|
06-Dec-2014 |
schwarze |
When opening mandoc.db fails, tell the user in which directory. Improving an unhelpful error message reported by millert@.
|
#
1.38 |
|
27-Nov-2014 |
schwarze |
Make makewhatis(8) understand .so links to .gz pages. Drop the FORM_GZ annotation in the mpages table; it is conceptually wrong because it ought to be in the mlinks table: An uncompressed .so link file can point to a compressed manual page file and vice versa. Besides, it is no longer needed because mparse_open() handles it all. Sprinkle some KNF while here.
|
#
1.37 |
|
18-Nov-2014 |
schwarze |
In man(1) mode, prefer file name matches over .Dt name matches over first .Nm entries over other NAME .Nm entries over SYNOPSIS .Nm entries. For example, this makes sure "man ypbind" does not return yp(8). Re-run "makewhatis" to profit from this change.
|
#
1.36 |
|
11-Nov-2014 |
schwarze |
In man(1) mode without -a, stop searching after the first manual tree that contained at least one match in order to not prefer mdoc(1) from ports over mdoc(7). As a bonus, this results in a speedup.
|
#
1.35 |
|
03-Sep-2014 |
schwarze |
If a manual page is installed gzip(1)ed, let makewhatis(8) take note in mandoc.db(5), such that man(1) -w and apropos(1) -w can report the correct filename. This is a prerequisite for letting apropos -a and man support gzip'ed manuals in the future, which doesn't work yet.
|
#
1.34 |
|
01-Sep-2014 |
schwarze |
In man(1) mode, change to the right directory before starting the parser, just like traditional man(1) does, such that .so links have a chance to work. After this point, we don't need the current directory for anything else before exit, so we don't need to worry about getting back and we can safely ignore failure. This lets man(1) find more Xenocara manuals, but not all of them yet. Other issues remain that need to be fixed, too.
|
#
1.33 |
|
27-Aug-2014 |
schwarze |
Add an implementation of man(1) into the /usr/bin/mandoc binary and provide a unified set of command line options for mandoc(1), man(1), apropos(1), and whatis(1), each option doing the same for all four. Not adding any completely new options, only extending exiting ones from one tool to the others. New options are: * apropos & whatis -acfkw (in the past, these were man(1) only) * apropos & whatis -a -IOTW (in the past, mandoc(1) only) * mandoc -ac (in the past, man(1) only) * man -IOTW (in the past, mandoc(1) only)
Before we can decide whether or not we want to replace src/usr.bin/man with this implementation, considerable bugfixing, testing, and performance measurements are needed, which i'd rather do in the tree than outside. Note that these bugs only affect the new man(1) mode, existing mandoc(1), apropos(1), and whatis(1) is fine.
The new functionality in mandoc(1), apropos(1), and whatis(1) is fully enabled. To play with the new man(1), you can try: # mv /usr/bin/man /usr/bin/oman # ln -s /usr/bin/mandoc /usr/bin/man
Positive feedback about the general direction from sthen@ and jmc@, and deraadt@ is not against it.
|
#
1.32 |
|
21-Aug-2014 |
schwarze |
Bugfix: make whatis(1) case-insensitive again. The traditional whatis(1) was case-insensitve and it's still documented that way, that but apparently got broken with or after the switch.
|
#
1.31 |
|
09-Aug-2014 |
schwarze |
mmap(2) requires MAP_PRIVATE ^ MAP_SHARED for flags; found by kristaps@ on Mac OS X
|
#
1.30 |
|
08-Aug-2014 |
schwarze |
Absurdly, the return value of sqlite3_column_text() is "const unsigned char *", which causes warnings with GCC on Linux. Explicitly cast to "const char *" to avoid this. Issue noticed by kristaps@.
|
Revision tags: OPENBSD_5_6_BASE
|
#
1.29 |
|
24-Jul-2014 |
schwarze |
Sort result pages first by section number, then by name. By moving the sort from cgi.c to mansearch.c, we get two advantages: Easier access to the data needed for sorting, in particular the section number, and the apropos(1) command line utility profits as well.
Feature requested by deraadt@.
|
#
1.28 |
|
12-Jul-2014 |
schwarze |
Fix whatis(1) to correctly match words instead of any substrings. While here, also provide an internal mode (MANSEARCH_MAN) to match complete names, to be used by man.cgi(8).
|
#
1.27 |
|
12-May-2014 |
espie |
adjust to ohash being in libutil now, and to the interface changes. fix potential integer overflows in memory allocation (mostly for pedagogical purposes, these are unlikely to overflow in practice) move the rest of lst.lib stuff into its own directory.
|
#
1.26 |
|
23-Apr-2014 |
schwarze |
Audit malloc(3)/calloc(3)/realloc(3) usage. * Change eight reallocs to reallocarray to be safe from overflows. * Change one malloc to reallocarray to be safe from overflows. * Change one calloc to reallocarray, no zeroing needed. * Change the order of arguments of three callocs (aesthetical).
|
#
1.25 |
|
23-Apr-2014 |
schwarze |
improve SQL style: avoid "SELECT *", be explicit in what columns we want; suggested by espie@.
|
#
1.24 |
|
20-Apr-2014 |
schwarze |
KNF: case (FOO): -> case FOO, remove /* LINTED */ and /* ARGSUSED */, remove trailing whitespace and blanks before tabs, improve some indenting; no functional change
|
#
1.23 |
|
17-Apr-2014 |
schwarze |
Garbage collect one pair of needless parentheses in SQL code generation; note this doesn't affect performance, SQLite generates the same byte code. While here, make the calls to exprspec() easier to understand.
|
#
1.22 |
|
16-Apr-2014 |
schwarze |
Rename the mpages.id column to mpages.pageid. There is no good reason to call this kid by a different name here than in all other tables. Easier to polish this now than after enabling.
|
#
1.21 |
|
16-Apr-2014 |
schwarze |
Pass the function flags SQLITE_UTF8 (because SQLITE_ANY is deprecated) and SQLITE_DETERMINISTIC when creating deterministic functions; best practice measure suggested by espie@ and jeremy@; as expected by jeremy@, no measurable effect on performance.
|
#
1.20 |
|
15-Apr-2014 |
schwarze |
At the end of mansearch(), fchdir() back to where we started from; this is cleaner and helps to not scatter gmon.out files all over the place when profiling.
|
#
1.19 |
|
11-Apr-2014 |
schwarze |
Further apropos(1) speed optimization was trickier than anticipated. Contrary to what i initially thought, almost all time is now spent inside sqlite3(3) routines, and i found no easy way calling less of them. However, sqlite(3) spends substantial time in malloc(3), and even more (twice that) in its immediate malloc wrapper, sqlite3MemMalloc(), keeping track of all individual malloc chunk sizes. Typically about 90% of the malloced memory is used for purposes of the pagecache.
By providing an mmap(3) MAP_ANON SQLITE_CONFIG_PAGECACHE, execution time decreases by 20-25% for simple (Nd and/or Nm) queries, 10-20% for non-NAME queries, and even apropos(1) resident memory size as reported by top(1) decreases by 20% for simple and by 60% for non-NAME queries. The new function, mansearch_setup(), spends no measurable time.
The pagesize chosen is optimal: * Substantially smaller pages yield no gain at all. * Larger pages provide no additional benefit and just waste memory.
The chosen number of pages in the cache is a compromise: * For simple queries, a handful of pages would suffice to get the full speed effect, at an apropos(1) resident memory size of about 2.0 MB. * For non-NAME queries, a large pagecache with 2k pages (2.5 MB) might gain a few more percent in speed, but at the expense of doubling the apropos(1) resident memory size for *all* queries. * The chosen number of 256 pages (330 kB) allows nearly full speed gain for all queries at the price of a 15% resident memory size increase.
|
#
1.18 |
|
10-Apr-2014 |
schwarze |
Next speed optimization step for the new apropos(1). Split manual names out of the common "keys" table into their own "names" table. This reduces standard apropos(1) search times (i.e. searching for names and descriptions only) by typically about 70% for the full /usr/share/man database. (Yes, that multiplies with the previous optimization step, so both together have reduced search times by a factor of more than six. I'm not done yet, expect more to come.)
Even with the minimal databases built with makewhatis(8) -Q, this step still reduces search times by 15-20%. For both cases, database sizes and build times hardly change (+/-2%).
|
#
1.17 |
|
09-Apr-2014 |
schwarze |
After careful gprof(1)ing of the new apropos(1), move the descriptions back from the keys table to the mpages table: I found a good way to still use them in searches, without complication of the code.
On my notebook, this reduces typical apropos(1) search times by about 40%, it reduces /usr/share/man database size by 6% in makewhatis(8) -Q mode and by 2% in standard mode (less overhead storing pointers to mpages), and it doesn't measurably change database build times (may even be going down by a percent or so because less data is being copied around in ohashes).
|
#
1.16 |
|
28-Mar-2014 |
schwarze |
Properly initialize malloc(3)ed memory. With this bug fix, partly unitialized memory could sometimes be returned, sometimes causing crashes by bogus free(3)s in apropos(1).
|
#
1.15 |
|
21-Mar-2014 |
schwarze |
avoid repetitive code for asprintf error handling
|
#
1.14 |
|
21-Mar-2014 |
schwarze |
The files mandoc.c and mandoc.h contained both specialised low-level functions used for multiple languages (mdoc, man, roff), for example mandoc_escape(), mandoc_getarg(), mandoc_eos(), and generic auxiliary functions. Split the auxiliaries out into their own file and header. While here, do some #include cleanup.
|
#
1.13 |
|
17-Mar-2014 |
schwarze |
in apropos(1) output, sort names and avoid multiple section numbers
|
Revision tags: OPENBSD_5_5_BASE
|
#
1.12 |
|
19-Jan-2014 |
schwarze |
Always compare arch case-insensitively.
|
#
1.11 |
|
19-Jan-2014 |
schwarze |
Get rid of the local keys table, use the new mansearch_const.c. No functional change.
|
#
1.10 |
|
06-Jan-2014 |
schwarze |
Remove the redundant "file" column from the "mlinks" table. The contents can easily be reconstructed from sec, arch, name, form. Shrinks the database by another 3% in standard mode and 9% in -Q mode.
|
#
1.9 |
|
06-Jan-2014 |
schwarze |
Drop Nd from the mpages table, it is still in the keys table. This shrinks the database in standard mode by 3%, in -Q mode by 9%, without loss of functionality.
|
#
1.8 |
|
05-Jan-2014 |
schwarze |
Remove the obsolete file name column from the mpages table. This column wasn't helpful because one manpage can have multiple MLINKS. Use the file name column in the mlinks table, instead.
|
#
1.7 |
|
05-Jan-2014 |
schwarze |
Remove the obsolete sec and arch columns from the mpages table. They were confusing because a manpage can have MLINKS in different sections and architectures.
|
#
1.6 |
|
05-Jan-2014 |
schwarze |
Reimplement apropos -s NUM -S ARCH EXPR by internally converting it to apropos \( EXPR \) -a 'sec~^NUM$' -a 'arch~^(ARCH|any)$' in preparation for removal of sec and arch from the mpage table.
Almost no functional change except for the following bonus: This also makes sure that for cross-section and cross-arch MLINKs, all of the following work: apropos -s 1 encrypt apropos -s 8 encrypt apropos -s 1 makekey apropos -s 8 makekey Before this commit, they don't, neither for the old makewhatis(8)/apropos(1) combo nor for the new one.
While here, print error messages about invalid regexps to stderr.
|
#
1.5 |
|
05-Jan-2014 |
schwarze |
Put section and architecture info into the keys table, in preparation for removing them from the mpages table, aiming for cleaner and more uniform interfaces. Database growth is below 4%, part of which will be reclaimed.
As a bonus, this allows searches like: ./obj/apropos An=kettenis -a arch=ppc ./obj/apropos An=kettenis -a sec~[^4]
|
#
1.4 |
|
04-Jan-2014 |
schwarze |
New implementation of complex search criteria using \(, \), -a because the old implementation got lost in the Berkeley to SQLite switch. Note that this is not just feature creep, but required for upcoming database format cleanup and simplification.
The following takes less than half a second on my Z61m ThinkPad: time ./obj/apropos An=kettenis -a \( Cd=vnet -o Xr=vbus \)
|
#
1.3 |
|
31-Dec-2013 |
schwarze |
Experimental feature to let apropos(1) show different keys than .Nd. This really takes us beyond what grep -R /usr/*/man/ can do because now you can search for pages by *one* criterion and then display the contents of *another* macro from those pages, like in $ apropos -O Ox Fa~wchar to get an impression how long wide character handling is available.
|
#
1.2 |
|
31-Dec-2013 |
schwarze |
Split buildnames() out of mansearch(); the latter function is getting too long and unwieldy, but will grow more code soon. No functional change.
|
#
1.1 |
|
31-Dec-2013 |
schwarze |
Replace the Berkeley-DB based mandocdb(8) by an SQLite3-based version, aiming for more flexible development and optimization options. Kristaps started this during the summer 2012, i did some very heavy bugfixing during t2k13 and finally, during the last few days, got it to a state where it is ripe for in-tree development. Beware, neither the user interfaces nor the database formats are expected to be stable just yet.
Will not be installed or activated until further discussion. No functional change to mandoc(1).
"As long as it remains off until we decide the cost, fine." deraadt@
|
#
1.62 |
|
22-Nov-2018 |
schwarze |
In apropos(1) output, stop sorting .Nm search results by name priorities (bits). The obscure feature wasn't documented and merely confused people - for example Edward Tomasz Napierala <trasz at FreeBSD>, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227408.
Smaller patch provided by Yuri Pankov <yuripv at FreeBSD>, but i'm also retiring the now unused "bits" member from struct manpage. Simplification is good.
|
#
1.61 |
|
19-Nov-2018 |
schwarze |
Improve POSIX compliance by making case-insensitive extended regular expressions the default in man(1) -k searches, also matching what the man-db package used by many Linux distributions does.
Originally requested by Wolfram Schneider <wosch at FreeBSD> via Yuri Pankov <yuripv at FreeBSD>.
Feedback and OK cheloha@, and no objections when shown on tech@.
Thanks to cheloha@ for pointing out that POSIX requires this behaviour and for the suggestion to explicitly say that *extended* regular expressions are used here.
While here, unify spelling of case-[in]sensitive, fix a typo, update the EXAMPLES, and add a STANDARDS section.
|
Revision tags: OPENBSD_6_2_BASE OPENBSD_6_3_BASE OPENBSD_6_4_BASE
|
#
1.60 |
|
22-Aug-2017 |
schwarze |
When finding a bogus database entry, print a meaningful warning and skip the entry. Issue reported by espie@.
|
#
1.59 |
|
02-Aug-2017 |
schwarze |
No longer use names that only occur in the SYNOPSIS section as names for man(1) lookup. For OpenBSD base and Xenocara, that functionality was never intended to be required, and i just fixed the last handful of offenders using it - not counting the horribly ill-designed interfaces engine(3) and lh_new(3) which are impossible to properly document in the first place.
Of course, apropos(1) and whatis(1) continue to use SYNOPSIS .Nm, .Fn, and .Fo macros, so "man -k ENGINE_get_load_privkey_function" still works.
This change also gets rid of a few bogus warnings "cross reference to self" which actually are *not* to self, like in yp(8).
This former functionality was intended to help third-party software in the ports tree and on non-OpenBSD systems containing manual pages with incomplete or corrupt NAME sections. But it turned out it did more harm than good, and caused more confusion than relief, specifically for third party manuals and for maintainers of mandoc-portable on other operating systems. So kill it. Problems reported, among others, by Yuri Pankov (illumos).
OK jmc@
|
#
1.58 |
|
19-Jul-2017 |
schwarze |
Prefer arch-dependent over arch-independent pages if the name priority, the section number, and all names match. Changes little on installed systems except the ordering of apropos(1) results, because we install base and Xenocara manuals in different trees, but fixes lookup of pages like apm(4) vs. apm(4/amd64) in man.cgi(8). Issue discovered by martian67 on freenode and reported via tj@.
|
#
1.57 |
|
01-Jul-2017 |
schwarze |
Basic reporting of .Xrs to manual pages that don't exist in the base system, inspired by mdoclint(1).
We are able to do this because (1) the -mdoc parser, the -Tlint validator, and the man(1) manual page lookup code are all in the same program and (2) the mandoc.db(5) database format allows fast lookup.
Feedback from, previous versions tested by, and OK jmc@.
A few features will be added to this in the tree, step by step.
|
#
1.56 |
|
17-May-2017 |
schwarze |
Do not warn if a database file does not exist because that is simply equivalent to an empty database. Suggested by ajacoutot@
|
#
1.55 |
|
18-Apr-2017 |
schwarze |
The apropos(1) manual still documents the unary -i operator for regular expression search terms, but it appears that somewhere along the way, the implementation got lost, so restore it. Bug found while investigating other reports from Gonzalo Tornaria.
|
#
1.54 |
|
18-Apr-2017 |
schwarze |
Fix previous: i forgot explicit NUL termination; noticed by Gonzalo <Tornaria at cmat dot edu dot uy>, thanks!
|
#
1.53 |
|
17-Apr-2017 |
schwarze |
Restore -kO Nd, -kO Nm, -kO sec, and -kO arch to working order. They got broken in the SQLite removal. As opposed to the rest of -kO, they are no longer very useful, but they are certainly not supposed to fail assertions. Issue reported by Gonzalo Tornaria <tornaria at cmat dot edu dot uy>.
|
Revision tags: OPENBSD_6_1_BASE
|
#
1.52 |
|
03-Mar-2017 |
schwarze |
A missing initialization could randomly cause regular expression searches to be case-insensitive that ought to be case sensitive. Found by jsg@ with scan-build.
|
#
1.51 |
|
01-Aug-2016 |
schwarze |
Remove the dependency on SQLite without loss of functionality. Drop the obsolete names_check() now that we deleted MLINKS. Run "doas makewhatis" after compiling and installing this.
Earlier version tested by jmc@ and jturner@; "commit it all" deraadt@ "commit and dodge" krw@
|
Revision tags: OPENBSD_6_0_BASE
|
#
1.50 |
|
09-Jul-2016 |
schwarze |
getopt(3) is declared in <unistd.h>, and <getopt.h> is not needed; from Joerg Sonnenberger via Thomas Klausner, NetBSD.
|
Revision tags: OPENBSD_5_9_BASE
|
#
1.49 |
|
08-Jan-2016 |
schwarze |
Prefer warn(3) over perror(3) at the few places where it was used. It is useful to see the program name. Suggested by Christos Zoulas (NetBSD).
|
#
1.48 |
|
26-Nov-2015 |
schwarze |
No point in trying to go on when elementary database operations like preparing queries or binding variables fail; that won't yield useful results anyway but may generate huge pointless error messages. Issue reported by deraadt@.
|
#
1.47 |
|
13-Oct-2015 |
schwarze |
Reduce the amount of code by moving the three copies of the ohash callback functions into one common place, preparing for the use of ohash for some additional purposes. No functional change.
|
#
1.46 |
|
11-Oct-2015 |
schwarze |
Finally use __progname, err(3) and warn(3). That's more readable and less error-prone than fumbling around with argv[0], fprintf(3), strerror(3), perror(3), and exit(3). It also shortens the code by 50 lines.
It's a bad idea to boycott good interfaces merely because standards committees ignore them. Instead, it's the job of the portable distribution to provide compatibility modules for archaic systems (like commercial Solaris) that still don't have them. Actually, the compat code for the portable distribution already exists and will be committed right after this.
|
#
1.45 |
|
06-Oct-2015 |
schwarze |
modernize style: "return" is not a function; ok cmp(1)
|
Revision tags: OPENBSD_5_8_BASE
|
#
1.44 |
|
01-Apr-2015 |
schwarze |
Let man(1) and apropos(1) work even when the current directory is unusable: Only change back to the current directory when the directory was changed before and the next path is relative. This is now more similar to what makewhatis(8) does. Issue reported by espie@.
|
#
1.43 |
|
27-Mar-2015 |
schwarze |
Parse the new man.conf(5) "output" directive. The next step will be to actually use the parsed data.
|
#
1.42 |
|
11-Mar-2015 |
schwarze |
When interpreting the -O argument as a macro name fails, fall back to showing Nd rather than not showing anything. Issue reported by jmc@.
|
Revision tags: OPENBSD_5_7_BASE
|
#
1.41 |
|
27-Feb-2015 |
schwarze |
When man(1) and apropos(1) look for a file man1/foo.1 but it's unavailable, fall back to glob(man1/foo.*), which is more like what old man(1) did. Do this both for file names from the database and for fs_lookup(). This is relevant because some ports install files like man1/xset.1x. Regression reported by patrick keshishian <pkeshish at gmail dot com>.
|
#
1.40 |
|
20-Jan-2015 |
schwarze |
Make the man(1) and apropos(1) options -s and -S much less expensive: Do not append an SQL clause looking into the large "keys" table. Instead, filter the result of the SQL query in buildnames() where equivalent data from the much smaller "mlinks" table is already available for free.
This is relevant because man(1) uses the equivalent of "-S ${MACHINE}" by default since main.c rev. 1.122, to make sure that manuals for the current architecture are shown. With many ports installed, this patch can speed up man(1) by a factor of more than a hundred.
Slowness reported by Theo Buehler <theo at math dot ethz dot ch>, thanks!
|
#
1.39 |
|
06-Dec-2014 |
schwarze |
When opening mandoc.db fails, tell the user in which directory. Improving an unhelpful error message reported by millert@.
|
#
1.38 |
|
27-Nov-2014 |
schwarze |
Make makewhatis(8) understand .so links to .gz pages. Drop the FORM_GZ annotation in the mpages table; it is conceptually wrong because it ought to be in the mlinks table: An uncompressed .so link file can point to a compressed manual page file and vice versa. Besides, it is no longer needed because mparse_open() handles it all. Sprinkle some KNF while here.
|
#
1.37 |
|
18-Nov-2014 |
schwarze |
In man(1) mode, prefer file name matches over .Dt name matches over first .Nm entries over other NAME .Nm entries over SYNOPSIS .Nm entries. For example, this makes sure "man ypbind" does not return yp(8). Re-run "makewhatis" to profit from this change.
|
#
1.36 |
|
11-Nov-2014 |
schwarze |
In man(1) mode without -a, stop searching after the first manual tree that contained at least one match in order to not prefer mdoc(1) from ports over mdoc(7). As a bonus, this results in a speedup.
|
#
1.35 |
|
03-Sep-2014 |
schwarze |
If a manual page is installed gzip(1)ed, let makewhatis(8) take note in mandoc.db(5), such that man(1) -w and apropos(1) -w can report the correct filename. This is a prerequisite for letting apropos -a and man support gzip'ed manuals in the future, which doesn't work yet.
|
#
1.34 |
|
01-Sep-2014 |
schwarze |
In man(1) mode, change to the right directory before starting the parser, just like traditional man(1) does, such that .so links have a chance to work. After this point, we don't need the current directory for anything else before exit, so we don't need to worry about getting back and we can safely ignore failure. This lets man(1) find more Xenocara manuals, but not all of them yet. Other issues remain that need to be fixed, too.
|
#
1.33 |
|
27-Aug-2014 |
schwarze |
Add an implementation of man(1) into the /usr/bin/mandoc binary and provide a unified set of command line options for mandoc(1), man(1), apropos(1), and whatis(1), each option doing the same for all four. Not adding any completely new options, only extending exiting ones from one tool to the others. New options are: * apropos & whatis -acfkw (in the past, these were man(1) only) * apropos & whatis -a -IOTW (in the past, mandoc(1) only) * mandoc -ac (in the past, man(1) only) * man -IOTW (in the past, mandoc(1) only)
Before we can decide whether or not we want to replace src/usr.bin/man with this implementation, considerable bugfixing, testing, and performance measurements are needed, which i'd rather do in the tree than outside. Note that these bugs only affect the new man(1) mode, existing mandoc(1), apropos(1), and whatis(1) is fine.
The new functionality in mandoc(1), apropos(1), and whatis(1) is fully enabled. To play with the new man(1), you can try: # mv /usr/bin/man /usr/bin/oman # ln -s /usr/bin/mandoc /usr/bin/man
Positive feedback about the general direction from sthen@ and jmc@, and deraadt@ is not against it.
|
#
1.32 |
|
21-Aug-2014 |
schwarze |
Bugfix: make whatis(1) case-insensitive again. The traditional whatis(1) was case-insensitve and it's still documented that way, that but apparently got broken with or after the switch.
|
#
1.31 |
|
09-Aug-2014 |
schwarze |
mmap(2) requires MAP_PRIVATE ^ MAP_SHARED for flags; found by kristaps@ on Mac OS X
|
#
1.30 |
|
08-Aug-2014 |
schwarze |
Absurdly, the return value of sqlite3_column_text() is "const unsigned char *", which causes warnings with GCC on Linux. Explicitly cast to "const char *" to avoid this. Issue noticed by kristaps@.
|
Revision tags: OPENBSD_5_6_BASE
|
#
1.29 |
|
24-Jul-2014 |
schwarze |
Sort result pages first by section number, then by name. By moving the sort from cgi.c to mansearch.c, we get two advantages: Easier access to the data needed for sorting, in particular the section number, and the apropos(1) command line utility profits as well.
Feature requested by deraadt@.
|
#
1.28 |
|
12-Jul-2014 |
schwarze |
Fix whatis(1) to correctly match words instead of any substrings. While here, also provide an internal mode (MANSEARCH_MAN) to match complete names, to be used by man.cgi(8).
|
#
1.27 |
|
12-May-2014 |
espie |
adjust to ohash being in libutil now, and to the interface changes. fix potential integer overflows in memory allocation (mostly for pedagogical purposes, these are unlikely to overflow in practice) move the rest of lst.lib stuff into its own directory.
|
#
1.26 |
|
23-Apr-2014 |
schwarze |
Audit malloc(3)/calloc(3)/realloc(3) usage. * Change eight reallocs to reallocarray to be safe from overflows. * Change one malloc to reallocarray to be safe from overflows. * Change one calloc to reallocarray, no zeroing needed. * Change the order of arguments of three callocs (aesthetical).
|
#
1.25 |
|
23-Apr-2014 |
schwarze |
improve SQL style: avoid "SELECT *", be explicit in what columns we want; suggested by espie@.
|
#
1.24 |
|
20-Apr-2014 |
schwarze |
KNF: case (FOO): -> case FOO, remove /* LINTED */ and /* ARGSUSED */, remove trailing whitespace and blanks before tabs, improve some indenting; no functional change
|
#
1.23 |
|
17-Apr-2014 |
schwarze |
Garbage collect one pair of needless parentheses in SQL code generation; note this doesn't affect performance, SQLite generates the same byte code. While here, make the calls to exprspec() easier to understand.
|
#
1.22 |
|
16-Apr-2014 |
schwarze |
Rename the mpages.id column to mpages.pageid. There is no good reason to call this kid by a different name here than in all other tables. Easier to polish this now than after enabling.
|
#
1.21 |
|
16-Apr-2014 |
schwarze |
Pass the function flags SQLITE_UTF8 (because SQLITE_ANY is deprecated) and SQLITE_DETERMINISTIC when creating deterministic functions; best practice measure suggested by espie@ and jeremy@; as expected by jeremy@, no measurable effect on performance.
|
#
1.20 |
|
15-Apr-2014 |
schwarze |
At the end of mansearch(), fchdir() back to where we started from; this is cleaner and helps to not scatter gmon.out files all over the place when profiling.
|
#
1.19 |
|
11-Apr-2014 |
schwarze |
Further apropos(1) speed optimization was trickier than anticipated. Contrary to what i initially thought, almost all time is now spent inside sqlite3(3) routines, and i found no easy way calling less of them. However, sqlite(3) spends substantial time in malloc(3), and even more (twice that) in its immediate malloc wrapper, sqlite3MemMalloc(), keeping track of all individual malloc chunk sizes. Typically about 90% of the malloced memory is used for purposes of the pagecache.
By providing an mmap(3) MAP_ANON SQLITE_CONFIG_PAGECACHE, execution time decreases by 20-25% for simple (Nd and/or Nm) queries, 10-20% for non-NAME queries, and even apropos(1) resident memory size as reported by top(1) decreases by 20% for simple and by 60% for non-NAME queries. The new function, mansearch_setup(), spends no measurable time.
The pagesize chosen is optimal: * Substantially smaller pages yield no gain at all. * Larger pages provide no additional benefit and just waste memory.
The chosen number of pages in the cache is a compromise: * For simple queries, a handful of pages would suffice to get the full speed effect, at an apropos(1) resident memory size of about 2.0 MB. * For non-NAME queries, a large pagecache with 2k pages (2.5 MB) might gain a few more percent in speed, but at the expense of doubling the apropos(1) resident memory size for *all* queries. * The chosen number of 256 pages (330 kB) allows nearly full speed gain for all queries at the price of a 15% resident memory size increase.
|
#
1.18 |
|
10-Apr-2014 |
schwarze |
Next speed optimization step for the new apropos(1). Split manual names out of the common "keys" table into their own "names" table. This reduces standard apropos(1) search times (i.e. searching for names and descriptions only) by typically about 70% for the full /usr/share/man database. (Yes, that multiplies with the previous optimization step, so both together have reduced search times by a factor of more than six. I'm not done yet, expect more to come.)
Even with the minimal databases built with makewhatis(8) -Q, this step still reduces search times by 15-20%. For both cases, database sizes and build times hardly change (+/-2%).
|
#
1.17 |
|
09-Apr-2014 |
schwarze |
After careful gprof(1)ing of the new apropos(1), move the descriptions back from the keys table to the mpages table: I found a good way to still use them in searches, without complication of the code.
On my notebook, this reduces typical apropos(1) search times by about 40%, it reduces /usr/share/man database size by 6% in makewhatis(8) -Q mode and by 2% in standard mode (less overhead storing pointers to mpages), and it doesn't measurably change database build times (may even be going down by a percent or so because less data is being copied around in ohashes).
|
#
1.16 |
|
28-Mar-2014 |
schwarze |
Properly initialize malloc(3)ed memory. With this bug fix, partly unitialized memory could sometimes be returned, sometimes causing crashes by bogus free(3)s in apropos(1).
|
#
1.15 |
|
21-Mar-2014 |
schwarze |
avoid repetitive code for asprintf error handling
|
#
1.14 |
|
21-Mar-2014 |
schwarze |
The files mandoc.c and mandoc.h contained both specialised low-level functions used for multiple languages (mdoc, man, roff), for example mandoc_escape(), mandoc_getarg(), mandoc_eos(), and generic auxiliary functions. Split the auxiliaries out into their own file and header. While here, do some #include cleanup.
|
#
1.13 |
|
17-Mar-2014 |
schwarze |
in apropos(1) output, sort names and avoid multiple section numbers
|
Revision tags: OPENBSD_5_5_BASE
|
#
1.12 |
|
19-Jan-2014 |
schwarze |
Always compare arch case-insensitively.
|
#
1.11 |
|
19-Jan-2014 |
schwarze |
Get rid of the local keys table, use the new mansearch_const.c. No functional change.
|
#
1.10 |
|
06-Jan-2014 |
schwarze |
Remove the redundant "file" column from the "mlinks" table. The contents can easily be reconstructed from sec, arch, name, form. Shrinks the database by another 3% in standard mode and 9% in -Q mode.
|
#
1.9 |
|
06-Jan-2014 |
schwarze |
Drop Nd from the mpages table, it is still in the keys table. This shrinks the database in standard mode by 3%, in -Q mode by 9%, without loss of functionality.
|
#
1.8 |
|
05-Jan-2014 |
schwarze |
Remove the obsolete file name column from the mpages table. This column wasn't helpful because one manpage can have multiple MLINKS. Use the file name column in the mlinks table, instead.
|
#
1.7 |
|
05-Jan-2014 |
schwarze |
Remove the obsolete sec and arch columns from the mpages table. They were confusing because a manpage can have MLINKS in different sections and architectures.
|
#
1.6 |
|
05-Jan-2014 |
schwarze |
Reimplement apropos -s NUM -S ARCH EXPR by internally converting it to apropos \( EXPR \) -a 'sec~^NUM$' -a 'arch~^(ARCH|any)$' in preparation for removal of sec and arch from the mpage table.
Almost no functional change except for the following bonus: This also makes sure that for cross-section and cross-arch MLINKs, all of the following work: apropos -s 1 encrypt apropos -s 8 encrypt apropos -s 1 makekey apropos -s 8 makekey Before this commit, they don't, neither for the old makewhatis(8)/apropos(1) combo nor for the new one.
While here, print error messages about invalid regexps to stderr.
|
#
1.5 |
|
05-Jan-2014 |
schwarze |
Put section and architecture info into the keys table, in preparation for removing them from the mpages table, aiming for cleaner and more uniform interfaces. Database growth is below 4%, part of which will be reclaimed.
As a bonus, this allows searches like: ./obj/apropos An=kettenis -a arch=ppc ./obj/apropos An=kettenis -a sec~[^4]
|
#
1.4 |
|
04-Jan-2014 |
schwarze |
New implementation of complex search criteria using \(, \), -a because the old implementation got lost in the Berkeley to SQLite switch. Note that this is not just feature creep, but required for upcoming database format cleanup and simplification.
The following takes less than half a second on my Z61m ThinkPad: time ./obj/apropos An=kettenis -a \( Cd=vnet -o Xr=vbus \)
|
#
1.3 |
|
31-Dec-2013 |
schwarze |
Experimental feature to let apropos(1) show different keys than .Nd. This really takes us beyond what grep -R /usr/*/man/ can do because now you can search for pages by *one* criterion and then display the contents of *another* macro from those pages, like in $ apropos -O Ox Fa~wchar to get an impression how long wide character handling is available.
|
#
1.2 |
|
31-Dec-2013 |
schwarze |
Split buildnames() out of mansearch(); the latter function is getting too long and unwieldy, but will grow more code soon. No functional change.
|
#
1.1 |
|
31-Dec-2013 |
schwarze |
Replace the Berkeley-DB based mandocdb(8) by an SQLite3-based version, aiming for more flexible development and optimization options. Kristaps started this during the summer 2012, i did some very heavy bugfixing during t2k13 and finally, during the last few days, got it to a state where it is ripe for in-tree development. Beware, neither the user interfaces nor the database formats are expected to be stable just yet.
Will not be installed or activated until further discussion. No functional change to mandoc(1).
"As long as it remains off until we decide the cost, fine." deraadt@
|
Revision tags: OPENBSD_6_2_BASE
|
#
1.60 |
|
22-Aug-2017 |
schwarze |
When finding a bogus database entry, print a meaningful warning and skip the entry. Issue reported by espie@.
|
#
1.59 |
|
02-Aug-2017 |
schwarze |
No longer use names that only occur in the SYNOPSIS section as names for man(1) lookup. For OpenBSD base and Xenocara, that functionality was never intended to be required, and i just fixed the last handful of offenders using it - not counting the horribly ill-designed interfaces engine(3) and lh_new(3) which are impossible to properly document in the first place.
Of course, apropos(1) and whatis(1) continue to use SYNOPSIS .Nm, .Fn, and .Fo macros, so "man -k ENGINE_get_load_privkey_function" still works.
This change also gets rid of a few bogus warnings "cross reference to self" which actually are *not* to self, like in yp(8).
This former functionality was intended to help third-party software in the ports tree and on non-OpenBSD systems containing manual pages with incomplete or corrupt NAME sections. But it turned out it did more harm than good, and caused more confusion than relief, specifically for third party manuals and for maintainers of mandoc-portable on other operating systems. So kill it. Problems reported, among others, by Yuri Pankov (illumos).
OK jmc@
|
#
1.58 |
|
19-Jul-2017 |
schwarze |
Prefer arch-dependent over arch-independent pages if the name priority, the section number, and all names match. Changes little on installed systems except the ordering of apropos(1) results, because we install base and Xenocara manuals in different trees, but fixes lookup of pages like apm(4) vs. apm(4/amd64) in man.cgi(8). Issue discovered by martian67 on freenode and reported via tj@.
|
#
1.57 |
|
01-Jul-2017 |
schwarze |
Basic reporting of .Xrs to manual pages that don't exist in the base system, inspired by mdoclint(1).
We are able to do this because (1) the -mdoc parser, the -Tlint validator, and the man(1) manual page lookup code are all in the same program and (2) the mandoc.db(5) database format allows fast lookup.
Feedback from, previous versions tested by, and OK jmc@.
A few features will be added to this in the tree, step by step.
|
#
1.56 |
|
17-May-2017 |
schwarze |
Do not warn if a database file does not exist because that is simply equivalent to an empty database. Suggested by ajacoutot@
|
#
1.55 |
|
18-Apr-2017 |
schwarze |
The apropos(1) manual still documents the unary -i operator for regular expression search terms, but it appears that somewhere along the way, the implementation got lost, so restore it. Bug found while investigating other reports from Gonzalo Tornaria.
|
#
1.54 |
|
18-Apr-2017 |
schwarze |
Fix previous: i forgot explicit NUL termination; noticed by Gonzalo <Tornaria at cmat dot edu dot uy>, thanks!
|
#
1.53 |
|
17-Apr-2017 |
schwarze |
Restore -kO Nd, -kO Nm, -kO sec, and -kO arch to working order. They got broken in the SQLite removal. As opposed to the rest of -kO, they are no longer very useful, but they are certainly not supposed to fail assertions. Issue reported by Gonzalo Tornaria <tornaria at cmat dot edu dot uy>.
|
Revision tags: OPENBSD_6_1_BASE
|
#
1.52 |
|
03-Mar-2017 |
schwarze |
A missing initialization could randomly cause regular expression searches to be case-insensitive that ought to be case sensitive. Found by jsg@ with scan-build.
|
#
1.51 |
|
01-Aug-2016 |
schwarze |
Remove the dependency on SQLite without loss of functionality. Drop the obsolete names_check() now that we deleted MLINKS. Run "doas makewhatis" after compiling and installing this.
Earlier version tested by jmc@ and jturner@; "commit it all" deraadt@ "commit and dodge" krw@
|
Revision tags: OPENBSD_6_0_BASE
|
#
1.50 |
|
09-Jul-2016 |
schwarze |
getopt(3) is declared in <unistd.h>, and <getopt.h> is not needed; from Joerg Sonnenberger via Thomas Klausner, NetBSD.
|
Revision tags: OPENBSD_5_9_BASE
|
#
1.49 |
|
08-Jan-2016 |
schwarze |
Prefer warn(3) over perror(3) at the few places where it was used. It is useful to see the program name. Suggested by Christos Zoulas (NetBSD).
|
#
1.48 |
|
26-Nov-2015 |
schwarze |
No point in trying to go on when elementary database operations like preparing queries or binding variables fail; that won't yield useful results anyway but may generate huge pointless error messages. Issue reported by deraadt@.
|
#
1.47 |
|
13-Oct-2015 |
schwarze |
Reduce the amount of code by moving the three copies of the ohash callback functions into one common place, preparing for the use of ohash for some additional purposes. No functional change.
|
#
1.46 |
|
11-Oct-2015 |
schwarze |
Finally use __progname, err(3) and warn(3). That's more readable and less error-prone than fumbling around with argv[0], fprintf(3), strerror(3), perror(3), and exit(3). It also shortens the code by 50 lines.
It's a bad idea to boycott good interfaces merely because standards committees ignore them. Instead, it's the job of the portable distribution to provide compatibility modules for archaic systems (like commercial Solaris) that still don't have them. Actually, the compat code for the portable distribution already exists and will be committed right after this.
|
#
1.45 |
|
06-Oct-2015 |
schwarze |
modernize style: "return" is not a function; ok cmp(1)
|
Revision tags: OPENBSD_5_8_BASE
|
#
1.44 |
|
01-Apr-2015 |
schwarze |
Let man(1) and apropos(1) work even when the current directory is unusable: Only change back to the current directory when the directory was changed before and the next path is relative. This is now more similar to what makewhatis(8) does. Issue reported by espie@.
|
#
1.43 |
|
27-Mar-2015 |
schwarze |
Parse the new man.conf(5) "output" directive. The next step will be to actually use the parsed data.
|
#
1.42 |
|
11-Mar-2015 |
schwarze |
When interpreting the -O argument as a macro name fails, fall back to showing Nd rather than not showing anything. Issue reported by jmc@.
|
Revision tags: OPENBSD_5_7_BASE
|
#
1.41 |
|
27-Feb-2015 |
schwarze |
When man(1) and apropos(1) look for a file man1/foo.1 but it's unavailable, fall back to glob(man1/foo.*), which is more like what old man(1) did. Do this both for file names from the database and for fs_lookup(). This is relevant because some ports install files like man1/xset.1x. Regression reported by patrick keshishian <pkeshish at gmail dot com>.
|
#
1.40 |
|
20-Jan-2015 |
schwarze |
Make the man(1) and apropos(1) options -s and -S much less expensive: Do not append an SQL clause looking into the large "keys" table. Instead, filter the result of the SQL query in buildnames() where equivalent data from the much smaller "mlinks" table is already available for free.
This is relevant because man(1) uses the equivalent of "-S ${MACHINE}" by default since main.c rev. 1.122, to make sure that manuals for the current architecture are shown. With many ports installed, this patch can speed up man(1) by a factor of more than a hundred.
Slowness reported by Theo Buehler <theo at math dot ethz dot ch>, thanks!
|
#
1.39 |
|
06-Dec-2014 |
schwarze |
When opening mandoc.db fails, tell the user in which directory. Improving an unhelpful error message reported by millert@.
|
#
1.38 |
|
27-Nov-2014 |
schwarze |
Make makewhatis(8) understand .so links to .gz pages. Drop the FORM_GZ annotation in the mpages table; it is conceptually wrong because it ought to be in the mlinks table: An uncompressed .so link file can point to a compressed manual page file and vice versa. Besides, it is no longer needed because mparse_open() handles it all. Sprinkle some KNF while here.
|
#
1.37 |
|
18-Nov-2014 |
schwarze |
In man(1) mode, prefer file name matches over .Dt name matches over first .Nm entries over other NAME .Nm entries over SYNOPSIS .Nm entries. For example, this makes sure "man ypbind" does not return yp(8). Re-run "makewhatis" to profit from this change.
|
#
1.36 |
|
11-Nov-2014 |
schwarze |
In man(1) mode without -a, stop searching after the first manual tree that contained at least one match in order to not prefer mdoc(1) from ports over mdoc(7). As a bonus, this results in a speedup.
|
#
1.35 |
|
03-Sep-2014 |
schwarze |
If a manual page is installed gzip(1)ed, let makewhatis(8) take note in mandoc.db(5), such that man(1) -w and apropos(1) -w can report the correct filename. This is a prerequisite for letting apropos -a and man support gzip'ed manuals in the future, which doesn't work yet.
|
#
1.34 |
|
01-Sep-2014 |
schwarze |
In man(1) mode, change to the right directory before starting the parser, just like traditional man(1) does, such that .so links have a chance to work. After this point, we don't need the current directory for anything else before exit, so we don't need to worry about getting back and we can safely ignore failure. This lets man(1) find more Xenocara manuals, but not all of them yet. Other issues remain that need to be fixed, too.
|
#
1.33 |
|
27-Aug-2014 |
schwarze |
Add an implementation of man(1) into the /usr/bin/mandoc binary and provide a unified set of command line options for mandoc(1), man(1), apropos(1), and whatis(1), each option doing the same for all four. Not adding any completely new options, only extending exiting ones from one tool to the others. New options are: * apropos & whatis -acfkw (in the past, these were man(1) only) * apropos & whatis -a -IOTW (in the past, mandoc(1) only) * mandoc -ac (in the past, man(1) only) * man -IOTW (in the past, mandoc(1) only)
Before we can decide whether or not we want to replace src/usr.bin/man with this implementation, considerable bugfixing, testing, and performance measurements are needed, which i'd rather do in the tree than outside. Note that these bugs only affect the new man(1) mode, existing mandoc(1), apropos(1), and whatis(1) is fine.
The new functionality in mandoc(1), apropos(1), and whatis(1) is fully enabled. To play with the new man(1), you can try: # mv /usr/bin/man /usr/bin/oman # ln -s /usr/bin/mandoc /usr/bin/man
Positive feedback about the general direction from sthen@ and jmc@, and deraadt@ is not against it.
|
#
1.32 |
|
21-Aug-2014 |
schwarze |
Bugfix: make whatis(1) case-insensitive again. The traditional whatis(1) was case-insensitve and it's still documented that way, that but apparently got broken with or after the switch.
|
#
1.31 |
|
09-Aug-2014 |
schwarze |
mmap(2) requires MAP_PRIVATE ^ MAP_SHARED for flags; found by kristaps@ on Mac OS X
|
#
1.30 |
|
08-Aug-2014 |
schwarze |
Absurdly, the return value of sqlite3_column_text() is "const unsigned char *", which causes warnings with GCC on Linux. Explicitly cast to "const char *" to avoid this. Issue noticed by kristaps@.
|
Revision tags: OPENBSD_5_6_BASE
|
#
1.29 |
|
24-Jul-2014 |
schwarze |
Sort result pages first by section number, then by name. By moving the sort from cgi.c to mansearch.c, we get two advantages: Easier access to the data needed for sorting, in particular the section number, and the apropos(1) command line utility profits as well.
Feature requested by deraadt@.
|
#
1.28 |
|
12-Jul-2014 |
schwarze |
Fix whatis(1) to correctly match words instead of any substrings. While here, also provide an internal mode (MANSEARCH_MAN) to match complete names, to be used by man.cgi(8).
|
#
1.27 |
|
12-May-2014 |
espie |
adjust to ohash being in libutil now, and to the interface changes. fix potential integer overflows in memory allocation (mostly for pedagogical purposes, these are unlikely to overflow in practice) move the rest of lst.lib stuff into its own directory.
|
#
1.26 |
|
23-Apr-2014 |
schwarze |
Audit malloc(3)/calloc(3)/realloc(3) usage. * Change eight reallocs to reallocarray to be safe from overflows. * Change one malloc to reallocarray to be safe from overflows. * Change one calloc to reallocarray, no zeroing needed. * Change the order of arguments of three callocs (aesthetical).
|
#
1.25 |
|
23-Apr-2014 |
schwarze |
improve SQL style: avoid "SELECT *", be explicit in what columns we want; suggested by espie@.
|
#
1.24 |
|
20-Apr-2014 |
schwarze |
KNF: case (FOO): -> case FOO, remove /* LINTED */ and /* ARGSUSED */, remove trailing whitespace and blanks before tabs, improve some indenting; no functional change
|
#
1.23 |
|
17-Apr-2014 |
schwarze |
Garbage collect one pair of needless parentheses in SQL code generation; note this doesn't affect performance, SQLite generates the same byte code. While here, make the calls to exprspec() easier to understand.
|
#
1.22 |
|
16-Apr-2014 |
schwarze |
Rename the mpages.id column to mpages.pageid. There is no good reason to call this kid by a different name here than in all other tables. Easier to polish this now than after enabling.
|
#
1.21 |
|
16-Apr-2014 |
schwarze |
Pass the function flags SQLITE_UTF8 (because SQLITE_ANY is deprecated) and SQLITE_DETERMINISTIC when creating deterministic functions; best practice measure suggested by espie@ and jeremy@; as expected by jeremy@, no measurable effect on performance.
|
#
1.20 |
|
15-Apr-2014 |
schwarze |
At the end of mansearch(), fchdir() back to where we started from; this is cleaner and helps to not scatter gmon.out files all over the place when profiling.
|
#
1.19 |
|
11-Apr-2014 |
schwarze |
Further apropos(1) speed optimization was trickier than anticipated. Contrary to what i initially thought, almost all time is now spent inside sqlite3(3) routines, and i found no easy way calling less of them. However, sqlite(3) spends substantial time in malloc(3), and even more (twice that) in its immediate malloc wrapper, sqlite3MemMalloc(), keeping track of all individual malloc chunk sizes. Typically about 90% of the malloced memory is used for purposes of the pagecache.
By providing an mmap(3) MAP_ANON SQLITE_CONFIG_PAGECACHE, execution time decreases by 20-25% for simple (Nd and/or Nm) queries, 10-20% for non-NAME queries, and even apropos(1) resident memory size as reported by top(1) decreases by 20% for simple and by 60% for non-NAME queries. The new function, mansearch_setup(), spends no measurable time.
The pagesize chosen is optimal: * Substantially smaller pages yield no gain at all. * Larger pages provide no additional benefit and just waste memory.
The chosen number of pages in the cache is a compromise: * For simple queries, a handful of pages would suffice to get the full speed effect, at an apropos(1) resident memory size of about 2.0 MB. * For non-NAME queries, a large pagecache with 2k pages (2.5 MB) might gain a few more percent in speed, but at the expense of doubling the apropos(1) resident memory size for *all* queries. * The chosen number of 256 pages (330 kB) allows nearly full speed gain for all queries at the price of a 15% resident memory size increase.
|
#
1.18 |
|
10-Apr-2014 |
schwarze |
Next speed optimization step for the new apropos(1). Split manual names out of the common "keys" table into their own "names" table. This reduces standard apropos(1) search times (i.e. searching for names and descriptions only) by typically about 70% for the full /usr/share/man database. (Yes, that multiplies with the previous optimization step, so both together have reduced search times by a factor of more than six. I'm not done yet, expect more to come.)
Even with the minimal databases built with makewhatis(8) -Q, this step still reduces search times by 15-20%. For both cases, database sizes and build times hardly change (+/-2%).
|
#
1.17 |
|
09-Apr-2014 |
schwarze |
After careful gprof(1)ing of the new apropos(1), move the descriptions back from the keys table to the mpages table: I found a good way to still use them in searches, without complication of the code.
On my notebook, this reduces typical apropos(1) search times by about 40%, it reduces /usr/share/man database size by 6% in makewhatis(8) -Q mode and by 2% in standard mode (less overhead storing pointers to mpages), and it doesn't measurably change database build times (may even be going down by a percent or so because less data is being copied around in ohashes).
|
#
1.16 |
|
28-Mar-2014 |
schwarze |
Properly initialize malloc(3)ed memory. With this bug fix, partly unitialized memory could sometimes be returned, sometimes causing crashes by bogus free(3)s in apropos(1).
|
#
1.15 |
|
21-Mar-2014 |
schwarze |
avoid repetitive code for asprintf error handling
|
#
1.14 |
|
21-Mar-2014 |
schwarze |
The files mandoc.c and mandoc.h contained both specialised low-level functions used for multiple languages (mdoc, man, roff), for example mandoc_escape(), mandoc_getarg(), mandoc_eos(), and generic auxiliary functions. Split the auxiliaries out into their own file and header. While here, do some #include cleanup.
|
#
1.13 |
|
17-Mar-2014 |
schwarze |
in apropos(1) output, sort names and avoid multiple section numbers
|
Revision tags: OPENBSD_5_5_BASE
|
#
1.12 |
|
19-Jan-2014 |
schwarze |
Always compare arch case-insensitively.
|
#
1.11 |
|
19-Jan-2014 |
schwarze |
Get rid of the local keys table, use the new mansearch_const.c. No functional change.
|
#
1.10 |
|
06-Jan-2014 |
schwarze |
Remove the redundant "file" column from the "mlinks" table. The contents can easily be reconstructed from sec, arch, name, form. Shrinks the database by another 3% in standard mode and 9% in -Q mode.
|
#
1.9 |
|
06-Jan-2014 |
schwarze |
Drop Nd from the mpages table, it is still in the keys table. This shrinks the database in standard mode by 3%, in -Q mode by 9%, without loss of functionality.
|
#
1.8 |
|
05-Jan-2014 |
schwarze |
Remove the obsolete file name column from the mpages table. This column wasn't helpful because one manpage can have multiple MLINKS. Use the file name column in the mlinks table, instead.
|
#
1.7 |
|
05-Jan-2014 |
schwarze |
Remove the obsolete sec and arch columns from the mpages table. They were confusing because a manpage can have MLINKS in different sections and architectures.
|
#
1.6 |
|
05-Jan-2014 |
schwarze |
Reimplement apropos -s NUM -S ARCH EXPR by internally converting it to apropos \( EXPR \) -a 'sec~^NUM$' -a 'arch~^(ARCH|any)$' in preparation for removal of sec and arch from the mpage table.
Almost no functional change except for the following bonus: This also makes sure that for cross-section and cross-arch MLINKs, all of the following work: apropos -s 1 encrypt apropos -s 8 encrypt apropos -s 1 makekey apropos -s 8 makekey Before this commit, they don't, neither for the old makewhatis(8)/apropos(1) combo nor for the new one.
While here, print error messages about invalid regexps to stderr.
|
#
1.5 |
|
05-Jan-2014 |
schwarze |
Put section and architecture info into the keys table, in preparation for removing them from the mpages table, aiming for cleaner and more uniform interfaces. Database growth is below 4%, part of which will be reclaimed.
As a bonus, this allows searches like: ./obj/apropos An=kettenis -a arch=ppc ./obj/apropos An=kettenis -a sec~[^4]
|
#
1.4 |
|
04-Jan-2014 |
schwarze |
New implementation of complex search criteria using \(, \), -a because the old implementation got lost in the Berkeley to SQLite switch. Note that this is not just feature creep, but required for upcoming database format cleanup and simplification.
The following takes less than half a second on my Z61m ThinkPad: time ./obj/apropos An=kettenis -a \( Cd=vnet -o Xr=vbus \)
|
#
1.3 |
|
31-Dec-2013 |
schwarze |
Experimental feature to let apropos(1) show different keys than .Nd. This really takes us beyond what grep -R /usr/*/man/ can do because now you can search for pages by *one* criterion and then display the contents of *another* macro from those pages, like in $ apropos -O Ox Fa~wchar to get an impression how long wide character handling is available.
|
#
1.2 |
|
31-Dec-2013 |
schwarze |
Split buildnames() out of mansearch(); the latter function is getting too long and unwieldy, but will grow more code soon. No functional change.
|
#
1.1 |
|
31-Dec-2013 |
schwarze |
Replace the Berkeley-DB based mandocdb(8) by an SQLite3-based version, aiming for more flexible development and optimization options. Kristaps started this during the summer 2012, i did some very heavy bugfixing during t2k13 and finally, during the last few days, got it to a state where it is ripe for in-tree development. Beware, neither the user interfaces nor the database formats are expected to be stable just yet.
Will not be installed or activated until further discussion. No functional change to mandoc(1).
"As long as it remains off until we decide the cost, fine." deraadt@
|