1.. Copyright (C) 2014-2020 Free Software Foundation, Inc.
2   Originally contributed by David Malcolm <dmalcolm@redhat.com>
3
4   This is free software: you can redistribute it and/or modify it
5   under the terms of the GNU General Public License as published by
6   the Free Software Foundation, either version 3 of the License, or
7   (at your option) any later version.
8
9   This program is distributed in the hope that it will be useful, but
10   WITHOUT ANY WARRANTY; without even the implied warranty of
11   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
12   General Public License for more details.
13
14   You should have received a copy of the GNU General Public License
15   along with this program.  If not, see
16   <http://www.gnu.org/licenses/>.
17
18Internals
19=========
20
21Working on the JIT library
22--------------------------
23Having checked out the source code (to "src"), you can configure and build
24the JIT library like this:
25
26.. code-block:: bash
27
28  mkdir build
29  mkdir install
30  PREFIX=$(pwd)/install
31  cd build
32  ../src/configure \
33     --enable-host-shared \
34     --enable-languages=jit,c++ \
35     --disable-bootstrap \
36     --enable-checking=release \
37     --prefix=$PREFIX
38  nice make -j4 # altering the "4" to however many cores you have
39
40This should build a libgccjit.so within jit/build/gcc:
41
42.. code-block:: console
43
44 [build] $ file gcc/libgccjit.so*
45 gcc/libgccjit.so:       symbolic link to `libgccjit.so.0'
46 gcc/libgccjit.so.0:     symbolic link to `libgccjit.so.0.0.1'
47 gcc/libgccjit.so.0.0.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
48
49Here's what those configuration options mean:
50
51.. option:: --enable-host-shared
52
53  Configuring with this option means that the compiler is built as
54  position-independent code, which incurs a slight performance hit,
55  but it necessary for a shared library.
56
57.. option:: --enable-languages=jit,c++
58
59  This specifies which frontends to build.  The JIT library looks like
60  a frontend to the rest of the code.
61
62  The C++ portion of the JIT test suite requires the C++ frontend to be
63  enabled at configure-time, or you may see errors like this when
64  running the test suite:
65
66  .. code-block:: console
67
68    xgcc: error: /home/david/jit/src/gcc/testsuite/jit.dg/test-quadratic.cc: C++ compiler not installed on this system
69    c++: error trying to exec 'cc1plus': execvp: No such file or directory
70
71.. option:: --disable-bootstrap
72
73  For hacking on the "jit" subdirectory, performing a full
74  bootstrap can be overkill, since it's unused by a bootstrap.  However,
75  when submitting patches, you should remove this option, to ensure that
76  the compiler can still bootstrap itself.
77
78.. option:: --enable-checking=release
79
80  The compile can perform extensive self-checking as it runs, useful when
81  debugging, but slowing things down.
82
83  For maximum speed, configure with ``--enable-checking=release`` to
84  disable this self-checking.
85
86Running the test suite
87----------------------
88
89.. code-block:: console
90
91  [build] $ cd gcc
92  [gcc] $ make check-jit RUNTESTFLAGS="-v -v -v"
93
94A summary of the tests can then be seen in:
95
96.. code-block:: console
97
98  jit/build/gcc/testsuite/jit/jit.sum
99
100and detailed logs in:
101
102.. code-block:: console
103
104  jit/build/gcc/testsuite/jit/jit.log
105
106The test executables are normally deleted after each test is run.  For
107debugging, they can be preserved by setting :envvar:`PRESERVE_EXECUTABLES`
108in the environment.  If so, they can then be seen as:
109
110.. code-block:: console
111
112  jit/build/gcc/testsuite/jit/*.exe
113
114which can be run independently.
115
116You can compile and run individual tests by passing "jit.exp=TESTNAME" to RUNTESTFLAGS e.g.:
117
118.. code-block:: console
119
120   [gcc] $ PRESERVE_EXECUTABLES= \
121             make check-jit \
122               RUNTESTFLAGS="-v -v -v jit.exp=test-factorial.c"
123
124and once a test has been compiled, you can debug it directly:
125
126.. code-block:: console
127
128   [gcc] $ PATH=.:$PATH \
129           LD_LIBRARY_PATH=. \
130           LIBRARY_PATH=. \
131             gdb --args \
132               testsuite/jit/test-factorial.c.exe
133
134Running under valgrind
135**********************
136
137The jit testsuite detects if :envvar:`RUN_UNDER_VALGRIND` is present in the
138environment (with any value).  If it is present, it runs the test client
139code under `valgrind <http://valgrind.org>`_,
140specifcally, the default
141`memcheck <http://valgrind.org/docs/manual/mc-manual.html>`_
142tool with
143`--leak-check=full
144<http://valgrind.org/docs/manual/mc-manual.html#opt.leak-check>`_.
145
146It automatically parses the output from valgrind, injecting XFAIL results if
147any issues are found, or PASS results if the output is clean.  The output
148is saved to ``TESTNAME.exe.valgrind.txt``.
149
150For example, the following invocation verbosely runs the testcase
151``test-sum-of-squares.c`` under valgrind, showing an issue:
152
153.. code-block:: console
154
155  $ RUN_UNDER_VALGRIND= \
156      make check-jit \
157        RUNTESTFLAGS="-v -v -v jit.exp=test-sum-of-squares.c"
158
159  (...verbose log contains detailed valgrind errors, if any...)
160
161                  === jit Summary ===
162
163  # of expected passes            28
164  # of expected failures          2
165
166  $ less testsuite/jit/jit.sum
167  (...other results...)
168  XFAIL: jit.dg/test-sum-of-squares.c: test-sum-of-squares.c.exe.valgrind.txt: definitely lost: 8 bytes in 1 blocks
169  XFAIL: jit.dg/test-sum-of-squares.c: test-sum-of-squares.c.exe.valgrind.txt: unsuppressed errors: 1
170  (...other results...)
171
172  $ less testsuite/jit/test-sum-of-squares.c.exe.valgrind.txt
173  (...shows full valgrind report for this test case...)
174
175When running under valgrind, it's best to have configured gcc with
176:option:`--enable-valgrind-annotations`, which automatically suppresses
177various known false positives.
178
179Environment variables
180---------------------
181When running client code against a locally-built libgccjit, three
182environment variables need to be set up:
183
184.. envvar:: LD_LIBRARY_PATH
185
186   `libgccjit.so` is dynamically linked into client code, so if running
187   against a locally-built library, ``LD_LIBRARY_PATH`` needs to be set
188   up appropriately.  The library can be found within the "gcc"
189   subdirectory of the build tree:
190
191  .. code-block:: console
192
193    $ file libgccjit.so*
194    libgccjit.so:       symbolic link to `libgccjit.so.0'
195    libgccjit.so.0:     symbolic link to `libgccjit.so.0.0.1'
196    libgccjit.so.0.0.1: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, not stripped
197
198.. envvar:: PATH
199
200  The library uses a driver executable for converting from .s assembler
201  files to .so shared libraries.  Specifically, it looks for a name
202  expanded from
203  ``${target_noncanonical}-gcc-${gcc_BASEVER}${exeext}``
204  such as ``x86_64-unknown-linux-gnu-gcc-5.0.0``.
205
206  Hence ``PATH`` needs to include a directory where the library can
207  locate this executable.
208
209  The executable is normally installed to the installation bindir
210  (e.g. /usr/bin), but a copy is also created within the "gcc"
211  subdirectory of the build tree for running the testsuite, and for ease
212  of development.
213
214.. envvar:: LIBRARY_PATH
215
216  The driver executable invokes the linker, and the latter needs to locate
217  support libraries needed by the generated code, or you will see errors
218  like:
219
220  .. code-block:: console
221
222    ld: cannot find crtbeginS.o: No such file or directory
223    ld: cannot find -lgcc
224    ld: cannot find -lgcc_s
225
226  Hence if running directly from a locally-built copy (without installing),
227  ``LIBRARY_PATH`` needs to contain the "gcc" subdirectory of the build
228  tree.
229
230For example, to run a binary that uses the library against a non-installed
231build of the library in LIBGCCJIT_BUILD_DIR you need an invocation of the
232client code like this, to preprend the dir to each of the environment
233variables:
234
235.. code-block:: console
236
237  $ LD_LIBRARY_PATH=$(LIBGCCJIT_BUILD_DIR):$(LD_LIBRARY_PATH) \
238    PATH=$(LIBGCCJIT_BUILD_DIR):$(PATH) \
239    LIBRARY_PATH=$(LIBGCCJIT_BUILD_DIR):$(LIBRARY_PATH) \
240      ./jit-hello-world
241  hello world
242
243Packaging notes
244---------------
245The configure-time option :option:`--enable-host-shared` is needed when
246building the jit in order to get position-independent code.  This will
247slow down the regular compiler by a few percent.  Hence when packaging gcc
248with libgccjit, please configure and build twice:
249
250  * once without :option:`--enable-host-shared` for most languages, and
251
252  * once with :option:`--enable-host-shared` for the jit
253
254For example:
255
256.. code-block:: bash
257
258  # Configure and build with --enable-host-shared
259  # for the jit:
260  mkdir configuration-for-jit
261  pushd configuration-for-jit
262  $(SRCDIR)/configure \
263    --enable-host-shared \
264    --enable-languages=jit \
265    --prefix=$(DESTDIR)
266  make
267  popd
268
269  # Configure and build *without* --enable-host-shared
270  # for maximum speed:
271  mkdir standard-configuration
272  pushd standard-configuration
273  $(SRCDIR)/configure \
274    --enable-languages=all \
275    --prefix=$(DESTDIR)
276  make
277  popd
278
279  # Both of the above are configured to install to $(DESTDIR)
280  # Install the configuration with --enable-host-shared first
281  # *then* the one without, so that the faster build
282  # of "cc1" et al overwrites the slower build.
283  pushd configuration-for-jit
284  make install
285  popd
286
287  pushd standard-configuration
288  make install
289  popd
290
291Overview of code structure
292--------------------------
293
294The library is implemented in C++.  The source files have the ``.c``
295extension for legacy reasons.
296
297* ``libgccjit.c`` implements the API entrypoints.  It performs error
298  checking, then calls into classes of the gcc::jit::recording namespace
299  within ``jit-recording.c`` and ``jit-recording.h``.
300
301* The gcc::jit::recording classes (within ``jit-recording.c`` and
302  ``jit-recording.h``) record the API calls that are made:
303
304   .. literalinclude:: ../../jit-common.h
305    :start-after: /* Recording types.  */
306    :end-before: /* End of recording types. */
307    :language: c++
308
309* When the context is compiled, the gcc::jit::playback classes (within
310  ``jit-playback.c`` and ``jit-playback.h``) replay the API calls
311  within langhook:parse_file:
312
313   .. literalinclude:: ../../jit-common.h
314    :start-after: /* Playback types.  */
315    :end-before: /* End of playback types. */
316    :language: c++
317
318   .. literalinclude:: ../../notes.txt
319    :lines: 1-
320
321Here is a high-level summary from ``jit-common.h``:
322
323.. include:: ../../jit-common.h
324  :start-after: This comment is included by the docs.
325  :end-before: End of comment for inclusion in the docs.  */
326
327.. _example-of-log-file:
328
329Another way to understand the structure of the code is to enable logging,
330via :c:func:`gcc_jit_context_set_logfile`.  Here is an example of a log
331generated via this call:
332
333.. literalinclude:: test-hello-world.exe.log.txt
334    :lines: 1-
335
336Design notes
337------------
338It should not be possible for client code to cause an internal compiler
339error.  If this *does* happen, the root cause should be isolated (perhaps
340using :c:func:`gcc_jit_context_dump_reproducer_to_file`) and the cause
341should be rejected via additional checking.  The checking ideally should
342be within the libgccjit API entrypoints in libgccjit.c, since this is as
343close as possible to the error; failing that, a good place is within
344``recording::context::validate ()`` in jit-recording.c.
345
346Submitting patches
347------------------
348Please read the contribution guidelines for gcc at
349https://gcc.gnu.org/contribute.html.
350
351Patches for the jit should be sent to both the
352gcc-patches@gcc.gnu.org and jit@gcc.gnu.org mailing lists,
353with "jit" and "PATCH" in the Subject line.
354
355You don't need to do a full bootstrap for code that just touches the
356``jit`` and ``testsuite/jit.dg`` subdirectories.  However, please run
357``make check-jit`` before submitting the patch, and mention the results
358in your email (along with the host triple that the tests were run on).
359
360A good patch should contain the information listed in the
361gcc contribution guide linked to above; for a ``jit`` patch, the patch
362shold contain:
363
364  * the code itself (for example, a new API entrypoint will typically
365    touch ``libgccjit.h`` and ``.c``, along with support code in
366    ``jit-recording.[ch]`` and ``jit-playback.[ch]`` as appropriate)
367
368  * test coverage
369
370  * documentation for the C API
371
372  * documentation for the C++ API
373
374A patch that adds new API entrypoints should also contain:
375
376  * a feature macro in ``libgccjit.h`` so that client code that doesn't
377    use a "configure" mechanism can still easily detect the presence of
378    the entrypoint.  See e.g. ``LIBGCCJIT_HAVE_SWITCH_STATEMENTS`` (for
379    a category of entrypoints) and
380    ``LIBGCCJIT_HAVE_gcc_jit_context_set_bool_allow_unreachable_blocks``
381    (for an individual entrypoint).
382
383  * a new ABI tag containing the new symbols (in ``libgccjit.map``), so
384    that we can detect client code that uses them
385
386  * Support for :c:func:`gcc_jit_context_dump_reproducer_to_file`.  Most
387    jit testcases attempt to dump their contexts to a .c file; ``jit.exp``
388    then sanity-checks the generated c by compiling them (though
389    not running them).   A new API entrypoint
390    needs to "know" how to write itself back out to C (by implementing
391    ``gcc::jit::recording::memento::write_reproducer`` for the appropriate
392    ``memento`` subclass).
393
394  * C++ bindings for the new entrypoints (see ``libgccjit++.h``); ideally
395    with test coverage, though the C++ API test coverage is admittedly
396    spotty at the moment
397
398  * documentation for the new C entrypoints
399
400  * documentation for the new C++ entrypoints
401
402  * documentation for the new ABI tag (see ``topics/compatibility.rst``).
403
404Depending on the patch you can either extend an existing test case, or
405add a new test case.  If you add an entirely new testcase: ``jit.exp``
406expects jit testcases to begin with ``test-``, or ``test-error-`` (for a
407testcase that generates an error on a :c:type:`gcc_jit_context`).
408
409Every new testcase that doesn't generate errors should also touch
410``gcc/testsuite/jit.dg/all-non-failing-tests.h``:
411
412  * Testcases that don't generate errors should ideally be added to the
413    ``testcases`` array in that file; this means that, in addition
414    to being run standalone, they also get run within
415    ``test-combination.c`` (which runs all successful tests inside one
416    big :c:type:`gcc_jit_context`), and ``test-threads.c`` (which runs all
417    successful tests in one process, each one running in a different
418    thread on a different :c:type:`gcc_jit_context`).
419
420    .. note::
421
422       Given that exported functions within a :c:type:`gcc_jit_context`
423       must have unique names, and most testcases are run within
424       ``test-combination.c``, this means that every jit-compiled test
425       function typically needs a name that's unique across the entire
426       test suite.
427
428  * Testcases that aren't to be added to the ``testcases`` array should
429    instead add a comment to the file clarifying why they're not in that
430    array. See the file for examples.
431
432Typically a patch that touches the .rst documentation will also need the
433texinfo to be regenerated.  You can do this with
434`Sphinx 1.0 <http://sphinx-doc.org/>`_ or later by
435running ``make texinfo`` within ``SRCDIR/gcc/jit/docs``.   Don't do this
436within the patch sent to the mailing list; it can often be relatively
437large and inconsequential (e.g. anchor renumbering), rather like generated
438"configure" changes from configure.ac.  You can regenerate it when
439committing to svn.
440