Determining the domain home directory of integrated weblogic server

In JDeveloper, the integrated weblogic server creates its runtime domain in a directory like $HOME/.jdeveloper/system12.1.3.0.41.140521.1008/DefaultDomain or $HOME/.jdeveloper/system/11.1.1.7.40.65.57.5/DefaultDomain. While $HOME and the fixed DefaultDomain are well known, it is not so easy to determine the actual name of the system directory – one could assume that there is only one and use system* as the name, but there could be more than one, especially if more than one JDeveloper version is installed at the same time and each of them had been creating its own WLS domain. What we actually need is to get the JDeveloper version since that is what is used in the system directory name. Assumed that $JDEV_HOME is the installation directory of the JDeveloper installation for which we would like to get its domain home directory (like /opt/Oracle/Middleware/Oracle_Home/jdeveloper), then the file $JDEV_HOME/jdev/bin/version.properties contains the full version number of JDeveloper, which is identical to the version number used in the system directory name:

$ grep VER_FULL $JDEV_HOME/jdev/bin/version.properties
VER_FULL=12.1.3.0.41.140521.1008

To extract the version number, we can use a simple sed script like

$ sed -n 's/^VER_FULL=\([0-9.]*\).*$/\1/p' $JDEV_HOME/jdev/bin/version.properties
12.1.3.0.41.140521.1008

(we need to make sure not to include the terminating line feed, hence we only collect digits and dots after the =). Finally we can now generate the complete directoy path to the domain directory:

$ JDEV_VER=`sed -n 's/^VER_FULL=\([0-9.]*\).*$/\1/p' $JDEV_HOME/jdev/bin/version.properties`
$ DOMAIN_HOME="${HOME}/.jdeveloper/system${JDEV_VER}/DefaultDomain"
$ ls $DOMAIN_HOME/servers
AdminServerTag  DefaultServer

This can be useful for any kind of automation tasks.

Using column aliases in a WHERE clause

Just wanted to answer a question on StackOverflow, where someone had a data model like this:

CREATE TABLE tvalues (cid NUMBER, cvalue NUMBER);
INSERT INTO tvalues VALUES(1, 200);
INSERT INTO tvalues VALUES(2, 1500);
INSERT INTO tvalues VALUES(3, 3200);
INSERT INTO tvalues VALUES(4, 4500);
COMMIT;
SELECT * FROM tvalues;

CID  CVALUE
------------
  1    200
  2   1500
  3   3200
  4   4500

The question was how to get the cid of the row so that a given value is between the value of the previous row and the value of the row for which to return the id (e.g. given value = 4000 should return id=4). Obviously, since in a relational database system, rows do not have an inherent order (there is no “next” or “previous” row), we need a sort criteria which I simply assumed to be the CID column. The solution I had in mind was to use an analytical function like LEAD() or LAG(). These functions allow to access the next or the previous row (see how the analytical function requires an ORDER definition, otherwise it would not know what the next or previous row is):

SELECT cid, LAG(cvalue, 1, 0) OVER(ORDER BY cid) cmin, cvalue cmax
FROM tvalues;

CID     CMIN        CMAX
--------------------------
  1        0         200
  2      200        1500
  3     1500        3200
  4     3200        4500

This returns a nice range-wise view of the data. Now the final solution should be easy: simply restrict the result set so that the given value is between the minimum and the maximum value:

SELECT cid, LAG(cvalue, 1, 0) OVER(ORDER BY cid) cmin, cvalue cmax
FROM tvalues
WHERE cmin < 4000 AND cmax > 4000;

However, this query returns an error message:

ORA-00904: "CMAX": invalid identifier

The reason is that it is generally not possible to use column aliases in the WHERE clause (or in any other part of the SELECT statement, with a few exceptions like ORDER BY). This also has nothing to do with the analytical functions – the following simple query does not work either:

SELECT cvalue avalue
FROM tvalues
WHERE avalue > 1000;

ORA-00904: "AVALUE": invalid identifier
00904. 00000 -  "%s: invalid identifier"

The reason is that the select list (the projection) is the last part which is processed in an SQL statement, hence the SQL parser does not yet know the column aliases when it processes other parts of the query, like the WHERE statement (btw, while investigating this, someone else posted a very similar answer to the original question). The final solution is easy: the query just needs to be wrapped into another query as a sub query:

SELECT * FROM (
   SELECT cid, LAG(cvalue, 1, 0) OVER(ORDER BY cid) cmin, cvalue cmax
   FROM tvalues
) WHERE cmin < 4000 AND cmax > 4000;

CID     CMIN        CMAX
--------------------------
4	3200	    4500

Tracing shared library calls with ltrace and latrace

Tracing shared library calls with ltrace

Similar to strace, which can be used to trace system calls, ltrace can be used to trace shared library calls. This can be very useful to get insight into the program flow of applications while analyzing an issue. Consider the following simple sample: hello.c:

include <stdio.h>

void helloFunc(const char* s) {
   printf(s);
}

hello.h:

void helloFunc(const char* s);

helloMain.c:

#include "hello.h"

int main() {
   helloFunc("Hello World, how are you?");
   return 0;
}

We create a shared library from hello.c:

$ gcc -fPIC --shared -o libhello.so hello.c

And we create the executable from helloMain.c which links against the library created before:

$ gcc -o hello helloMain.c -lhello -L.

We can now use ltrace to see which library calls are executed:

$ export LD_LIBRARY_PATH=.
$ ltrace ./hello 
__libc_start_main(0x40069d, 1, 0x7fffe2f3b778, 0x4006c0 <unfinished ...>
helloFunc(0x400744, 0x7fffe2f3b778, 0x7fffe2f3b788, 0)              = 25
Hello World, how are you?+++ exited (status 0) +++

One drawback with ltrace is that it only traces calls from the executable to the libraries the executable is linked against – it does not trace calls between libraries! Hence, the call to the printf() function (which itself resides in the libc shared library) is not shown in the output. Also, there is no option to include the library name in the output for each of the called functions.

Tracing shared library calls with latrace

Especially for larger applications, a better alternative to ltrace is latrace which uses the LD_AUDIT feature from the libc library (available from libc 2.4 onward). On Ubuntu, it can be installed with

$ sudo apt-get install latrace

When using latrace with the sample program from above, there are two important differences:

  • First, latrace also traces library calls between shared libraries, so the output includes the printf call executed from our own shared library.
  • Second, the shared library name which contains the symbol is printed after each function call:

$ latrace ./hello
10180     _dl_find_dso_for_object [/lib64/ld-linux-x86-64.so.2]  
10180     __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]  
10180       helloFunc [./libhello.so]  
10180         printf [/lib/x86_64-linux-gnu/libc.so.6]  
10180       __tls_get_addr [/lib64/ld-linux-x86-64.so.2]  
10180       __cxa_finalize [/lib/x86_64-linux-gnu/libc.so.6]  
Hello World, how are you?
./hello finished - exited, status=0

By default, function parameters are not shown, but this can be enabled with the -A option:

$ latrace -A ./hello
10190     _dl_find_dso_for_object [/lib64/ld-linux-x86-64.so.2]  
10190     __libc_start_main(main = 0x40069d, argc = 1, ubp_av = 0x7fffccd35248, auxvec = 0x4006c0, init = 0x400730, fini = 0x7f3d16b21560, rtld_fini = 0x7fffccd35238) [/lib/x86_64-linux-gnu/libc.so.6] {
10190       helloFunc [./libhello.so]  
10190         printf(format = "Hello World, how are you?") [/lib/x86_64-linux-gnu/libc.so.6] {
Hello World, how are you?10190         } printf = 25
10190       __tls_get_addr [/lib64/ld-linux-x86-64.so.2]  
10190       __cxa_finalize(ptr = 0x7f3d1650b030) [/lib/x86_64-linux-gnu/libc.so.6] {
10190       } __cxa_finalize = void

With -A, ltrace uses some configuration files at /etc/latrace.d to define the parameter format for a set of well-known functions such as printf. We can see in the output that, even though helloFunc() takes a parameter, this parameter is not shown in the output (since it is not defined in the configuration files). We can use the -a option to specify our own argument definition file. An argument definition file is essentially a header file which defines the prototypes for all functions for which the arguments should be displayed in the output. On Ubuntu, the default argument definition files are stored at /etc/latrace.d/headers/, and there is a master file /etc/latrace.d/headers/latrace.h which includes the other header files. We can use the same approach in our own definition file, by first including the master file from /etc/latrace.d/headers/latrace.h and then add the prototypes for each function we want to trace. For our ssample above, the file could look like

#include "/etc/latrace.d/headers/latrace.h"
void helloFunc(const char* s);

Assumed this file is called mylatrace.h, wen can now use the -a option to pass the file to latrace:

$ latrace -a mylatrace.h -A ./hello
10801     _dl_find_dso_for_object [/lib64/ld-linux-x86-64.so.2]  
10801     __libc_start_main(main = 0x40069d, argc = 1, ubp_av = 0x7fff4b89fb58, auxvec = 0x4006c0, init = 0x400730, fini = 0x7f1927ea1560, rtld_fini = 0x7fff4b89fb48) [/lib/x86_64-linux-gnu/libc.so.6] {
10801       helloFunc(s = "Hello World, how are you?") [./libhello.so] {
10801         printf(format = "Hello World, how are you?") [/lib/x86_64-linux-gnu/libc.so.6] {
10801         } printf = 25
10801       } helloFunc = void
10801       __tls_get_addr [/lib64/ld-linux-x86-64.so.2]  
Hello World, how are you?10801       __cxa_finalize(ptr = 0x7f192788b030) [/lib/x86_64-linux-gnu/libc.so.6] {
10801       } __cxa_finalize = void
./hello finished - exited, status=0

As you can see in the output, we now also see that helloFunc() is called with one parameter s which is set to “Hello World, how are you?”.

Understanding free and top memory statistics [Update]

Both top and free can be used to gather basic information about memory usage, but each of them reports the statistics in a slightly different way which might not be directly obvious. An example output of free, using the -m switch to report numbers in MiB instead of KiB, is as follows:

Lets ignore the last line (Swap – it simply shows the total swap space and how much from that swap space is allocated and how much is still free) and focus on physical memory: The first three numbers in the Mem: line are straight forward: the “total” column shows the total physical memory available (most likely, this system has 8 GiB installed and uses a part of it for its graphics device, hence the “total” column shows less than 8 GiB). The “used” column shows the amount of memory which is currently in use, and the “free” column shows the amount which is still available. Then, there are the “buffers” and “cached” columns – they show how much from the “used” memory is really used for buffers and caches. Buffers and caches is memory which the kernel uses for temporary data – if an application requires more memory, and there is no memory “free” anymore, the kernel can still use this temporary memory and assign it to application processes (probably resulting in lower I/O performance since there is not as much cache memory available now). Finally, there is the “+/- buffers/cache” line: This might look strange first, but what it does is that it also reports the “used” and “free” memory, without the buffers and caches – as said above, buffer and cache memory is dynamic and can be assigned to an application process, if required. Hence, the “+/- buffers/cache” line actually shows the memory which is used by and available for processes. The following diagram shows the memory allocation from the sample above:

top returns almost the same information, in a slightly different layout (note that the numbers are somewhat different since some time has elapsed between the execution of the two commands):

The main difference is that it does not directly show the “used” and “free” memory without the buffers – but this can be easily calculated. Another thing which looks strange is that the amount of “cached” memory is shown in the “Swap” line – however, it has nothing to do with swap, probably it has been put there to use the available screen area as efficient as possible.

Update: procps >= 3.3.10

Starting with procps 3.3.10, the output of free has changed which might cause some confusion. I came across this through a question on StackOverflow: Linux “free -m”: Total, used and free memory values don’t add up. Essentially, free does not show the “+/- buffers/cache” line anymore, but instead shows an “available” column which is taken from the MemAvailable metric which has been introduced with kernel 3.14. See https://www.kernel.org/doc/Documentation/filesystems/proc.txt for a complete description:

MemAvailable: An estimate of how much memory is available for starting new applications, without swapping. Calculated from MemFree, SReclaimable, the size of the file LRU lists, and the low watermarks in each zone. The estimate takes into account that the system needs some page cache to function well, and that not all reclaimable slab will be reclaimable, due to items being in use. The impact of those factors will vary from system to system.
The new format of the free output looks like this:

The main difference is that the “buff/cache” values are not part of “Used” anymore, but counted separately. Hence, the total memory is calculated as “used + buff/cache + free”:

Since the “available” value is an estimation which considers some system specific factors, it can not directly be calculated from the other values which are shown by free.