• Usage is limited to personal, non-commercial use.
• “It may be used freely on up to 2 computers, and up to a maximum of 16 cores per computer, but is not licensed for use on any cloud, only personal computers.”"

The binary and complete terms of usage are here.

What fby does is aggregate values from one list based on groups defined in another, parallel, list. For example, suppose we have one list of cities and another list with a few temperature samples for each. We can use fby to calculate the minimum temperature sample for each city, and then replicate those values at each position for each corresponding city:

q)city:`NY`NY`LA`SF`LA`SF`NY
q)temp:32 31 75 69 70 68 12
q)(min;temp) fby city
12 12 70 68 70 68 12
q)

Thus, NY’s minimum temperature, 12, appears at every index in fby‘s output where `NY appears in city.

At the core of fby (like by) is group, which organizes the distinct values in a list into a dictionary mapping those values to their indices:

q)city:`NY`NY`LA`SF`LA`SF`NY
q)group city
NY| 0 1 6
LA| 2 4
SF| 3 5
q)

We can group the temperatures for each city together by indexing temp with the value of the grouped city dictionary:

q)grouped: value group city
q)temp[grouped]
32 31 12
75 70
69 68
q)

Note that the result of indexing temp with grouped is a nested list with the same shape as grouped. This is a general principle: the result of an indexing operation has the shape of the index.

Now we can apply an aggregation function to each of the temperature groups:

q)min each temp[grouped]
12 70 68
q)

We’re almost there. The real trick of fby is placing each aggregation result into a new list so that each element has the correct value per the grouping list. We can use @ (functional amend) to get the job done (see also the functional apply/amend faq):

q)@[temp; grouped; :; min each temp[grouped] ]
12 12 70 68 70 68 12
q)

The real fby is just slightly more complicated to ensure that the first argument to @ has the correct type.

A table is a reference to a (column) dictionary.

The internal representation of a table is nearly identical to that of a dictionary. We can use the flip command to create a table from a dictionary:

q)t: ([] a: 1 2 3; b: 4 5 6; c: 7 8 9)
q)d: `a`b`c ! (1 2 3; 4 5 6; 7 8 9)
q)t ~ flip d
1b
q)

In fact, when a dictionary is flip‘ed, the underlying core data structure remains untouched. The table itself is a simple, small object that refers to the original dictionary. Using .Q.w, we can measure how much more memory a table takes than the corresponding dictionary:

q).Q.w[]`used         // memory usage baseline
112992j
q)x:`a`b`c!3 3#til 9  // create a small dictionary
q).Q.w[]`used
113424j
q)x:flip x
q).Q.w[]`used         // memory usage delta is
113456j               // just 32 more bytes
q)

No matter how large the underlying dictionary is, creating a table is fast and still takes only 32 bytes:

q)x:`a`b`c!3 100000#til 10
q).Q.w[]`used
1686192j
q)x:flip x
q).Q.w[]`used
1686224j          // 32 = 1686224 - 1686192
q)

Now, Let’s examine how a keyed table is related to a dictionary. We start by creating a simple keyed table:

q)t: ([] a: 1 2 3; b: 4 5 6; c: 7 8 9)
q)keyedTable: `a`b xkey t
q)keyedTable
a b| c
---| -
1 4| 7
2 5| 8
3 6| 9
q)keys keyedTable
`a`b
q)

Since keyedTable is a table, one might expect it to have the same type as t but instead, q presents the following surprise:

q)type t
98h                  // type table as expected
q)type keyedTable
99h                  // *NOT* 98h
q)

Type 99h is the type number for dictionaries. If keyedTable really is a dictionary, we should be able to extract its key and value:

q)key keyedTable
a b
---
1 4
2 5
3 6
q)value keyedTable
c
-
7
8
9
q)

Indeed, keyedTable is a dictionary – one that holds unkeyed tables for both its key and its value:

q)type key keyedTable
98h
q)type value keyedTable
98h
q)

This suggests that we can create a keyed table by using the ! (dict) operator with two unkeyed tables:

q)(key keyedTable)!(value keyedTable)
a b| c
---| -
1 4| 7
2 5| 8
3 6| 9
q)

Lastly, joining the two flipped tables brings us back to the original dict.

q)(flip key keyedTable),(flip value keyedTable)
a| 1 2 3
b| 4 5 6
c| 7 8 9
q)

For more information, see Creating dictionaries and tables from C and , (join).

q)phandle: hopen `:serverhost:5012
q)phandle (function; arg1; arg2 ..)
..
q)hclose phandle
q)

To invoke a function in another q process (i.e., a function that already exists in that q process), you need the following:

1. a process handle to the other q process,
2. the name of the pre-existing function in the other process, and
3. the parameters you want to pass.

Then all you need to do is to apply the process handle to a list whose first element is the function name (as a symbol) and whose remaining elements are the arguments.

For example, let’s start one q process, which will be our server, listening on TCP port 5012. In our server, we’ll define a function square (we’ll make the background color different for the server to make it easier to distinguish from the client):

$ q -p 5012             // server
..
q)\p                    // display the listening port
5012
q)square: {x * x}
q)square 3
9
q)

Next, we’ll start up another q process (the client), create a process handle connected to the first q process (the server), and request the first process to compute square 5:

$ q                     // client
..
q)phandle: hopen `:localhost:5012
q)phandle (`square; 5)  // remote execution
25
q)

To call a function with more parameters, simply add them to the end of the list. We’ll demonstrate by defining a 2-argument function on the server that calculates the length of a right triangle’s hypotenuse:

q)hypotenuse: {sqrt sum square x,y}   // server
q)hypotenuse[3;4]
5f
q)

q)phandle (`hypotenuse; 5; 12)    // client
13f
q)

What if the function you’re calling doesn’t take any parameters? For example, we’ll define a function in the server called serverTime that returns the local time according to the server:

q)serverTime: {[] .z.T}    // server
q)serverTime[]
11:51:34.762
q)

You can’t pass zero parameters over IPC:

q)phandle enlist `serverTime  // a list with just
'length                       // the function name
q)                            // is not allowed

However, you can pass whatever you like, and it won’t matter:

q)phandle `serverTime`ignored
11:52:28.537
q)phandle (`serverTime; ())    // () is often used
11:52:40.923
q)phandle (`serverTime; ::)    // so is ::
11:52:47.338
q)

See :: (generic null).

So far, all of our examples involved a client invoking a predefined function on the server. You can also pass a function defined on the client to be executed on the server. To see this, let’s define a global variable on the server:

q)SERVERGLOBAL: 47     // server
q)

Now, on the client, we’ll define a function called getSERVERGLOBAL to retrieve the value of SERVERGLOBAL on the server. Instead of passing the name of the function (i.e., `getSERVERGLOBAL), we pass the function’s value:

q)getSERVERGLOBAL: {[] SERVERGLOBAL}    // client
q)getSERVERGLOBAL[]
'SERVERGLOBAL                   // not defined here
q)phandle (getSERVERGLOBAL; ::) // note the missing `
47
q)

Notice that this operation does not cause getSERVERGLOBAL to become defined on the server:

q)getSERVERGLOBAL                 // server
'getSERVERGLOBAL
q)

This technique can be used to run built-in functions and anonymous functions (aka lambdas) on the server as well:

q)phandle (+; 2; 4)                // client
6
q)phandle ({til SERVERGLOBAL - x}; 42)
0 1 2 3 4
q)

There is one more way to convey code to the server to run: you can pass the code in a string.

q)phandle "SERVERGLOBAL + 4"      // client
51
q)

We prefer passing a function over a string, because – especially as the expression to be passed gets more complex – it’s easier to read.

q)container: 1 2 3
q)100 * container
100 200 300
q)

Sometimes we are interested in only a subset of elements from a container:

q)container: til 10
q)container
0 1 2 3 4 5 6 7 8 9
q)container[where 0 = container mod 2]
0 2 4 6 8
q)

Sometimes, however, you need to update particular elements of a structure while leaving the remaining elements unchanged. That’s what functional apply and amend are for; they transform specific elements of a container without touching the others. The variations are distinguished by 3 choices:

1. @ or .

Which operator is used, @ or ., determines the interpretation of the indices used to select the elements to transform.

2. container or name

The first argument is either the value of a container or the name of a global variable referring to a container. In the former case, a new object is returned; in the latter, the global variable is modified and its name is returned.

3. monadic or dyadic function

If the transformation requires additional information beyond that contained in each element itself, that is accomplished by using a dyadic function and supplying the additional information in a fourth argument to the operator.

For a detailed discussion, please see the following faqs:

Functional @ (apply)

Functional . (apply)

Lastly, note that there is another pair of overloads for @ and . – each with three arguments – called protected execution, which are invoked when the first argument to @ or . is a function or projection; protected execution is discussed in another faq.

1. @[container; indices; function]
2. @[container; indices; function; second_args]

(Note that there is another overload for @ with three arguments, called protected execution, which is invoked when the first argument to @ is a function or projection; protected execution is discussed in another faq.)

In the 3-argument case, q applies function to those elements of container specified by indices, leaving the rest of container alone. In other words, the behavior of 3-argument @ (apply) resembles that of the following function:

Although this model breaks down when the first argument is a global variable name, it is very helpful in understanding what is going on even in that case. Let’s look at a couple of simple examples to clarify this:

q)list: 1 + til 8
q)list
1 2 3 4 5 6 7 8
q)@[list; 0; neg]         / negate element 0
-1 2 3 4 5 6 7 8
q)list                    / list remains unmodified
1 2 3 4 5 6 7 8
q)@[list; 1 3 4; {x * x}] / square elements 1, 3, and 4
1 4 3 16 25 6 7 8
q)

The diagram below illustrates the last example above:

Because the new container has the same type and shape as the original, function must return the same type as its argument, unless the container is a mixed list (i.e., type 0):

q)@[list; 4 5 6 7; {x div 2}] / list is an int list
1 2 3 4 2 3 3 4
q)@[list; 4 5 6 7; {x % 2}]   / but % returns float
'type
q)mixed_list: 1 2, `3`4
q)type mixed_list
0h
q)@[mixed_list; 0 3; string] / anything goes with type 0
,"1"
2
`3
,"4"
q)

Next, we’ll consider the second case listed at the top of this faq. When function is dyadic and a fourth argument is supplied, @ (apply) behaves like the following function:

The basic idea is still the same, i.e., to transform selected elements while leaving the rest of the container intact. The difference is that, instead of function being modified by each, it is modified by ‘(each-both) (also see this faq on each-both and multivalent each), so that the selected elements of container are paired up with the corresponding elements from second_args:

q)@[list; 1 3 5 7; +; 10 20 30 40]
1 12 3 24 5 36 7 48
q)@[list; (1 3 5 7; 0 2 4 6); *; 2 -2]
-2 4 -6 8 -10 12 -14 16
q)

One function often used with 4-argument @ (apply), is : (amend). We can replace selected elements with a certain value (or values) as follows:

q)list: 1 + til 8
q)@[list; 1 3 5 7; :; 47]
1 47 3 47 5 47 7 47
q)@[list; 1 3 5 7; :; 10 20 30 40]
1 10 3 20 5 30 7 40
q)

It might appear that the above example exposes a flaw in our model, apply_dyadic, of @ (apply). However, recall that, when modified by an adverb, : (amend) does not modify its first argument:

q):'[list 1 2 3; 20 30 40]
20 30 40                     / :' returns its 2nd argument
q)list                       / without modifying its 1st
1 2 3 4 5 6 7 8
q)

Where our models do break down is when, as alluded to earlier, the first argument is the name of global variable referring to a container, rather than the value of a container. In that case, the mechanics of the operation are the same, but original container is modified, and the return value is the name:

q)list: 1 + til 8
q)@[`list; 0; neg]
`list
q)list
-1 2 3 4 5 6 7 8
q)

This behavior is handy for writing functions of your own that work in both scenarios – either creating a new value or modifying one in place – like the following:

We can also use @ to apply a function to selected entries in a dictionary:

q)dictionary: `a`b`c`d ! `1`2, 3 4
q)dictionary
a| `1
b| `2
c| 3
d| 4
q)@[dictionary; `a`b; {"I"$ string x}]
a| 1
b| 2
c| 3
d| 4
q)

Tables can be transformed as well. Consider the following table:

q)t: ([] col1: `foo`bar`baz; col2: 5 10 15f)
q)t
col1 col2
---------
foo  5
bar  10
baz  15
q)

Recall that each row of a table is a dictionary:

q)t 0
col1| `foo
col2| 5f
q)

Thus, we can use @ (apply) to accomplish an update in the following manner:

q)update col2 - 4 from t where i > 0
col1 col2
---------
foo  5
bar  6
baz  11
q)@[t; 1 2; {[row] row[`col2] -: 4; row}]
col1 col2
---------
foo  5
bar  6
baz  11
q)

We could also obtain the previous result using nested calls to @ (apply):

q)@[t; 1 2; @[; `col2; -[; 4]]]
col1 col2
---------
foo  5
bar  6
baz  11
q)

Since we can index tables using column names, we can double all of the entries in col2 as follows:

q)@[t; `col2; 2*]
col1 col2
---------
foo  10
bar  20
baz  30
q)

1. .[container; indices; function]
2. .[container; indices; function; second_args]

Note that there is another overload for . (apply) with three arguments, called protected execution, which is invoked when the first argument to . (apply) is a function or projection; protected execution is discussed in another faq.

In the 3-argument case, q applies function to those elements of container specified by indices, leaving the rest of container alone.

The behavior of . (apply) is very similar to that of @ (apply) (which is described in another faq). The only difference between @ (apply) and . (apply) is that the indices, in the case of . (apply), are applied at depth along the dimensions of the container. The following code behaves like . (apply) for a two-dimensional container:

To explore the different treatment of the indices argument between @ (apply) and . (apply), we’ll consider the simplest, multi-dimensional container: a two-dimensional list, aka a matrix.

q)matrix: 0N 4 # til 16
q)matrix
0  1  2  3
4  5  6  7
8  9  10 11
12 13 14 15
q)

We can use simple indexing of matrix with both @ (index) and . (index) to illustrate the difference:

q)@[matrix; 0]           // fetch row 0
0 1 2 3
q)@[matrix; 0 0]         // fetch matrix[0;0] fails.  why?
0 1 2 3                  // @ cannot index more
0 1 2 3                  // than one dimension
q)
q).[matrix; 0 0]         // dot indexes "at depth"
0
q)

Now let’s apply a function to selected elements from matrix using . (apply):

q).[matrix; (0; ::); 100+]
100 101 102 103
4   5   6   7
8   9   10  11
12  13  14  15
q).[matrix; 0 0; 100+]
100 1  2  3
4   5  6  7
8   9  10 11
12  13 14 15
q).[matrix; (::; 0); 100+]
100 1  2  3
104 5  6  7
108 9  10 11
112 13 14 15
q)

Notice that we used :: in order to elide a dimension from the indices.

We can also use . (apply) with dictionaries of lists:

q)dict: `a`b ! (1 2 3; 4 5 6)
q).[dict; (`a; 0); 50*]
a| 50 2 3
b| 4  5 6
q).[dict; (::; 1); 50*]
a| 1 100 3
b| 4 250 6
q)

The general case of applying a function at depth via functional . (apply) is not implemented for tables:

q)t: ([] a: `foo`bar; b: 5 10f)
q).[t; (0; `b); %[; 5]]
'nyi
q).[t; (`b; 0); %[; 5]]
'nyi
q)

The . form of apply has another trick up its sleeve: the empty list index. When the second parameter to . (apply) is (), the entire container is passed to function in a single invocation, as the following example demonstrates:

q).[matrix; (); 0N!]
(0 1 2 3;4 25 36 7;8 81 100 11;12 13 14 15)
0  1  2   3
4  25 36  7
8  81 100 11
12 13 14  15
q)

When using . (apply) in this manner, we can return anything from function; the type and shape do not matter:

q).[matrix; (); {"hello"}]
"hello"
q)

Returning to the second case, when function is dyadic, . (apply) takes a fourth argument (named second_args in our example), and each indexed element of container is paired with the corresponding element of second_args. This means that second_args must conform to indices (or be an atom). The following function expresses this behavior:

(Also see this faq on each-both and multivalent each.)

We can demonstrate how this works by altering the center square of matrix:

q).[matrix; (1 2; 1 2); *; (10 20; 100 200)]
0  1   2    3
4  50  120  7
8  900 2000 11
12 13  14   15
q)

We can categorize the ways in which q handles vectors into the following three groups:

1. Treat the vector as a single entity.

2. Perform an operation on each element of a vector.

3. Perform an operation on each set of corresponding elements from multiple vectors of the same length.

We use ‘ (each-both) to ask q to apply a function in the third manner.

Let’s review a few examples to distinguish the three forms of vector handling described above. Consider the , (join) operator, which takes two arguments and concatenates them. When used by itself, , (join) treats a vector as a single entity:

q)1 , 2
1 2
q)1 2 3 , 10
1 2 3 10
q)1 , 10 20 30
1 10 20 30
q)1 2 , 10 20 30
1 2 10 20 30
q)

Notice that the lengths of the two vectors to be joined doesn’t matter; in fact, they don’t have to be vectors.

Next, as an example of applying the same operation to each element of a vector, we can use \: (each-left) to append the letter “e” to each element in a vector of strings:

q)("car"; "far"; "mar") ,\: "e"
"care"
"fare"
"mare"
q)

Lastly, we’ll use ‘ (each-both) to join corresponding elements from two vectors of the same length (like zip from Python or Haskell):

q)1 2 3 ,' 10 20 30
1 10
2 20
3 30
q)

As an aside, the use of ‘ (each-both) is not needed for many built-in functions in q, because those functions (sometimes referred to as atomic functions in kx documentation) automatically assume that this is the desired behavior when presented with same-length vectors as arguments. The + operator is a good example:

q)1 2 3 + 10 20 30
11 22 33
q)

Even a user defined function, if that function’s body is exclusively made up of applications of atomic functions, does not require the use of ‘ (each-both) to display this behavior:

q)atomic: {[x; y] (x * x) + y * y}
q)atomic'[1 2 3; 10 20 30]
101 404 909
q)atomic[1 2 3; 10 20 30]  // ' not required
101 404 909
q){[x; y] atomic[x; y]}[1 2 3; 10 20 30]
101 404 909
q)

The last example shows that such user defined atomic functions are truly atomic in the eyes of q.

The following code simulates the behavior of ‘ (each-both) – on functions of two arguments – by creating a new function (that acts on vectors) from the function passed as an argument:

However, ‘ (each-both) doesn’t stop at two lists, which is why it is sometimes called multivalent each. Suppose we need to generate a bunch of html hyperlinks for a web page we are creating on-the-fly (perhaps in a custom http POST handler). Each link needs to have its own styling, so we want to add a distinct class attribute to each anchor tag.

The .h namespace includes numerous functions for generating HTML. The one we need is the .h.htac (html tag with attributes and closing tag) function:

q).h.htac[`a; `class`href ! ("myclass"; "kdbfaq.com"); "kdbfaq"]
"<a class=\"myclass\" href=\"kdbfaq.com\">kdbfaq</a>"
q)

Let’s wrap .h.htac (html tag with attributes and closing tag) up in a function to create a styled link:

We’ll store the information we need to generate the links in a table:

q)links
name        url                    class
----------------------------------------
"kx"        "http://kx.com"        "c1"
"Wikipedia" "http://wikipedia.com" "c2"
"kdbfaq"    "http://kdbfaq.com"    "c3"
q)

Now we can use ‘ (multivalent each) to generate all the links easily:

q)styled_link'[links `name; links `url; links `class]
"<a class=\"c1\" href=\"http://kx.com\">kx</a>"
"<a class=\"c2\" href=\"http://wikipedia.com\">Wikipedia</a>"
"<a class=\"c3\" href=\"http://kdbfaq.com\">kdbfaq</a>"
q)

Or, alternately, we can combine ‘ (multivalent each) with . (apply) for a shorter, more general approach to applying a function to each row in a table:

q)styled_link .' flip value flip links
"<a class=\"c1\" href=\"http://kx.com\">kx</a>"
"<a class=\"c2\" href=\"http://wikipedia.com\">Wikipedia</a>"
"<a class=\"c3\" href=\"http://kdbfaq.com\">kdbfaq</a>"
q)

On more wrinkle: sometimes ‘ (each) is used where \: (each-left) or /: (each-right) would be more explicit. For instance, our earlier example in which we appended the letter “e” to each element of a list of strings could have been written as follows:

q)("car"; "far"; "mar") ,' "e"
"care"
"fare"
"mare"
q)

We find the original, more explicit form, quicker and easier to read.

Note: This applies to bash, and has not been tested in csh, ksh, etc.

We can invoke arbitrary external programs from q using the system command. For example:

q)system "echo Hello, world!"
"Hello, world!"
q)

External programs often signal errors by returning a non-zero exit code, which is stored in the shell variable ?:

$ ls nonexistent_file
ls: nonexistent_file: No such file or directory
$ echo $?
1
$ foo
-bash: foo: command not found
$ echo $?
127
$

When an external program called from system in q runs into a situation like this, q detects the non-zero exit code from the child process and raises a signal, ‘os:

q)system "ls nonexistent_file"
ls: nonexistent_file: No such file or directory
'os
q)

When trying out ideas at the console as in the above example, the full details of the error are available on the screen. Getting those error details programmatically is a bit trickier, however. Consider the following function, in which we use protected execution to invoke an error handler when the ‘os signal is raised:

use_protected_execution: {[]
  : @[system;
  "ls nonexistent_file";
  {[error_info] -1 "error is ", error_info;}];
}

The error handler in use_protected_execution knows only that an operating system error occurred. None of the error details are available within the code:

q)use_protected_execution[]
ls: nonexistent_file: No such file or directory  // invisible
error is os
q)

We can improve the situation somewhat by appending “; echo $?” to the command:

q)result: system "ls nonexistent_file; echo $?"
q)result
,"1"    // actually a list of strings
q)

This enables us to distinguish error codes as follows:

distinguish_error_codes: {[]
 result: system "ls nonexistent_file; echo $?";
 exit_code: "I" $ last result;
 $[0   = exit_code;
   ; // everything's alright
   1   = exit_code;
   ; // minor problem
   2   = exit_code;
   ; // major problem
   127 = exit_code;
   ; // command not found
   / else
   // unhandled exit code
]}

What we really want, though, is to capture the output sent by the failed command to STDERR. You might expect the usual shell redirection syntax to do the job:

q)result: system "ls nonexistent_file 2>&1; echo $?"
ls: nonexistent_file: No such file or directory
q)result
,"1"
q)

No such luck. The trick is to put parentheses around the whole thing:

q)result: system "(ls nonexistent_file 2>&1; echo $?)"
q)result
"ls: nonexistent_file: No such file or directory"
,"1"
q)

We can wrap all this up in a handy function that, instead of raising os, raises the descriptive message output to STDERR by the child process:

call_external: {[command]
  result: system "(", command, " 2>&1; echo $?)";
  exit_code: "I" $ last result;
  if [0 < exit_code;
      ' raze -1 _ result];
  -1 _ result}

To experiment, let’s construct two dictionaries, one with atomic keys and one with list keys.

q)dict1: `keya`keyb`keyc ! 10 20 30
q)dict1                // dictionary with atomic keys
keya| 10
keyb| 20
keyc| 30
q)dict2: (`keya1`keya2; `keyb1`keyb2; `keyc1`keyc2) ! 1 2 3
q)dict2                // dictionary with list keys
keya1 keya2| 1
keyb1 keyb2| 2
keyc1 keyc2| 3
q)dict1[`keya]         // index by atom key
10
q)dict2[`keya1`keya2]  // index by list key
1
q)

Note how dict1 contains a keys made of exclusively of atoms, and dict2 contains keys that are lists.

Now let’s consider a dictionary whose keys are both atoms and lists:

q)dict3: (`keya1; `keyb1`keyb2; `keyc1`keyc2) ! 1 2 3
q)key dict3
`keya1                // The first key is an atomic symbol,
`keyb1`keyb2          // while the rest of the keys are
`keyc1`keyc2          // lists of symbols.
q)dict3[`keya1]       // The first key indexes fine,
1
q)dict3[`keyb1`keyb2] // but the remaining keys are broken;
0N 0N
q)dict3[`keyc1`keyc2] // they are treated as multiple keys.
0N 0N
q)

Even though the keys are present they cannot be found by a dictionary lookup. Reverse lookup does work:

q)dict3
`keya1      | 1    // ok
`keyb1`keyb2| 2    // broken
`keyc1`keyc2| 3    // broken
q)dict3 ? 2
`keyb1`keyb2
q)dict3 ? 3
`keyc1`keyc2
q)

Analogous behavior can be observed when searching a list that contains both atoms and lists:

q)list: key dict3
q)list ? `keya1
0q)list ? `keyb1`keyb2
3 3
q)

Always remember that a dictionary lookup is essentially the following:

q)lookup: {[dict; thekey] value[dict] @ key[dict] ? thekey}
q)

Sometimes types in q behave differently depending on the data contained within. Take a look at this related faq to read more.

Let’s test with the following two dictionaries, sfdict and fsdict:

q)sfdict: `symbol`float ! (`abc; 123.4)  // type sym is first
q)fsdict: `float`symbol ! (123.4; `abc)  // type float is first
q)sfdict
symbol| `abc
float | 123.4
q)fsdict
float | 123.4
symbol| `abc
q)

Note that while both dictionaries contain the same key-value pairs, because dictionaries are an ordered list of key value pairs, the above two dictionaries do not match.

q)sfdict = fsdict
symbol| 1
float | 1
q)sfdict ~ fsdict
0b
q)

A dictionary search returns a null value when the key sought is not found:

q)null sfdict[`badkey]
1b
q)null fsdict[`badkey]
1b
q)

Getting a null back from a failed lookup is expected. On the other hand, many are surprised to find that the two nulls returned for the above two lookups differ:

q)sfdict[`badkey] ~ fsdict[`badkey]
0b
q)sfdict[`badkey]
`                    // null symbol
q)fsdict[`badkey]
0n                   // null float
q)

The types of the returned nulls are different according to the types of the first values in sfdict and fsdict. This is the same behavior observed when indexing a mixed list with an out-of-range index:

q)(`a; `b; 1f; 2f)[5]
`
q)(1f; 2f; `a; `b)[5]
0n
q)

There’s a similar “first element wins” behavior with the atomicity of dictionary keys. see this faq on dictionary indexing to learn more.

Normally, the : (amend) function is used to assign a value to a name:

q)foo: 47
q)foo
47
q)

Like other built-in functions that take two arguments, : (amend) can be called using either infix notation (as above) or function call notation:

q):[foo; 747]
q)foo
747
q)

Note that : (amend) displays its special semantics (which it shares with assignments in all strict languages) of not evaluating its first argument when that argument is simply a name, regardless of whether : (amend) is invoked infix or functionally.

q)delete bar from `.    / make sure bar is not defined
q):[bar; 42]
q)bar
42
q)

On the other hand, : (amend) does evaluate its first argument when that argument is an expression, but : (amend) still produces l-values when it does so:

q)list: 0 1 2 3
q)list[0]: 47
q)list
47 1 2 3
q):[list 1 2 3; 10 20 30]
q)list
47 10 20 30
q)

Based on the previous examples, you might be tempted create a projection from : (amend) and a left-hand side, but it turns out you can’t:

q):[foo]
'foo
q)

It is possible to close the first argument to : (amend), but the resulting projection is no longer an assignment; it is an identity function:

q)foo: 42
q)(:[foo])[47]
47
q)foo
42
q)

In fact, the value attached to the first argument of : (amend) is irrelevant:

q)(:[`xyzzy])[47]
47
q)

Closing the second argument will cause : (amend) to return the same value always:

q)f: :[; 3]     / bind the 2nd argument
q)f 18          / no matter what we pass
3
q)f "hello"     / f will always return 3
3
q)

The other case when : (amend) does not perform assignment is when it is modified by an adverb:

q)list1: 0 1 2 3
q)list2: 4 5 6 7
q)(list1; list2) :\: 10 20 30 40   / we get the rhs
10 20 30 40                        / once for each list
10 20 30 40
q)list1                            / neither list
0 1 2 3
q)list2                            / has been modified
4 5 6 7
q)list2 :/: 0 1 2 3
0 1 2 3
q)list1[0 1 2 3] :' 10 20 30 40
10 20 30 40
q)

q)table: ([] x: 10 ? "abc"; y: 10 ? til 10)
q)table
x y
---
a 7
a 7
b 1
b 9
a 1
a 0
c 8
b 8
c 3
c 1
q)

To sort by column x, instead of ORDER BY, we write `x xasc table:

q)`x xasc table // sort by column x
x y
---
a 7
a 7
a 1
a 0
b 1
b 9
b 8
c 8
c 3
c 1
q)

The application of xdesc is similar.

In addition, to sort by multiple columns, instead of ORDER BY y, x or ORDER BY y, x DESC , we pass the list of column names as the left argument to xasc or xdesc, respectively. For example,

q)`y`x xdesc select from table where y > 5
x y
---
b 9
c 8
b 8
a 7
a 7
q)

Don’t confuse xasc and xdesc with asc and desc, which operate on a vector instead of a table. Read more about sorting vectors in this related faq.

q)(`hh`mm`ss $ .z.T), `int $ .z.T mod 1000
16 49 2 233
q)

You can extract the hours, minutes, and seconds from a time by passing `hh, `mm, and `ss, respectively, as left arguments to $ (cast):

q)now: .z.T
q)now
16:49:02.233
q)`hh $ now
16
q)`mm $ now
49
q)`ss $ now
2
q)`hh`mm`ss $ now
16 49 2
q)

Getting the milliseconds from a time is slightly less obvious. Times (type -19) are represented internally by q as 32-bit integers; typically the value counts the number of milliseconds since midnight, but it can also represent a span of time. We can cast freely back and forth between the two types and the values are preserved:

q)`time $ 0
00:00:00.000
q)`int $ 00:00:00.000
0q)`time $ 24 * 60 * 60 * 1000
24:00:00.000
q)`int $ 24:00:00.000
86400000
q)`int $ now
60542233
q)`time $ 60542233
16:49:02.233
q)

Not only is the internal representation of time simply an integer, we can mix integers and times in integer arithmetic operations, and the result is always a time:

q)01:00:00.000 + 00:01:00.000
01:01:00.000
q)01:00:00.000 + 60000
01:01:00.000
q)01:00:00.000 * 4
04:00:00.000
q)now - 01:30:20.123
15:18:42.110
q)

By using div and mod, then, we have an alternative means to calculate the components of a time:

q)now div 3600000 // milliseconds per hour
00:00:00.016
q)now mod 1000    // just the milliseconds, please
00:00:00.233
q)

Although extracting milliseconds from a time while keeping the time type (as in the second example above) is sometimes useful, we normally want to get back these components of a time as integers, so let’s cast it:

q)`int $ now mod 1000
233
q)

By the way, there is a shortcut for getting hours, minutes, and seconds from global variables that hold times: dot notation.

q)x: .z.T
q)x.hh, x.mm, x.ss
16 49 2
q)

However, we rarely use global variables to hold time values.

In general you can parse strings to some particular type by passing the uppercase form of the corresponding type character as the left argument to $ (cast).

q can parse dates in YYYYMMDD, YYYY-MM-DD, or YYYY.MM.DD format:

q)"D"$ ("20070809"; "2007-08-09"; "2007.08.09")
2007.08.09 2007.08.09 2007.08.09
q)

Times must have the form HH:MM:SS.mmm, but you can shorten them:

q)"T"$ ("07:08:09.010"; "07:08:09"; "07:08"; "07")
07:08:09.010 07:08:09.000 07:08:00.000 07:00:00.000
q)

In addition, “T”$ will tolerate a missing zero for hours (and hours only) less than 10:

q)"T"$ "7:08:09"
07:08:09.000
q)

“Z”$ expects (as in XML) that dates and times are joined with a T:

q)"Z"$ "2007-08-09T07:08:09.101"
2007.08.09T07:08:09.101
q)

“Z”$ usually treats the text analogously to “D”$ and “T”$; that is, the date must be complete, but the time may have its finer points elided:

q)"Z"$ "2007"
0Nz
q)"Z"$ "20070809"
2007.08.09T00:00:00.000
q)

Note, however, the “Z”$ – unlike “T”$ – requires that hours less than 10 be prepended with zero:

q)"Z"$ "20070809T7:08:09"
0Nz
q)

You can also parse months with “M”$, minutes with “U”$, and seconds with “V”$:

q)"M"$ ("2007-08"; "2007.08"; "200708")
2007.08 2007.08 2007.08m
q)"U"$ "07:08:09.010"
07:08
q)"V"$ "07:08:09.010"
07:08:09
q)

Make sure you don’t pass any extra text to “M”$, though:

q)"M"$ "2007.08.01"
2007.01m
q)

There are two more type characters for parsing temporal values. “N”$ and “P”$ are for parsing nanosecond-precise time spans and datetimes, respectively:

q)"N"$ "12D07:08:09.123456789"
12D07:08:09.123456789
q)"P"$ "2007-08-09T07:08:09.123456789"
2007.08.09D07:08:09.123456789
q)

You may have discovered that “N”$ can parse floating point numbers, as well, but beware – depending on the range of the input, you might not get what you expect:

q)"N"$ "0.123456789"
0D00:00:00.123456789
q)"N"$ "1.123456789"
0D01:00:00.123456789
q)"N"$ "12.123456789"
0D12:00:00.123456789
q)"N"$ "24.123456789"
1D00:00:00.123456789
q)"N"$ "48.123456789"
2D00:00:00.123456789
q)"N"$ "4800.123456789"
2D00:00:00.123456789
q)

Stick with strings that represent durations in the way q does, and you’ll steer clear of surprises.

Lastly, if you find yourself with a text full of times expressed as seconds (and, optionally, fractions thereof) since the Unix epoch, “P”$, and “Z”$ can parse them, too:

q)unix_time: first system "date +%s"
q)unix_time
"1311635286"
q)"Z"$ unix_time
2011.07.25T23:08:06.000
q)"P"$ unix_time , ".123456789"
2011.07.25D23:08:06.123456789
q)

count list
or
count each column_name
to get the length of a list-valued field in a query.

The length of any list can be found using the count function; the only complication occurs with certain applications of count in queries. We’ll illustrate with a table of simulated trade data:

q)trade
time         sym price size
---------------------------
09:30:17.623 glo 26.79 3200
09:30:26.602 ojb 28.17 1600
09:30:33.657 fjc 91.31 5400
09:30:40.884 knb 59.7  2900
09:31:19.256 apl 1.26  7900
..
q)

The following query uses ( count ) to return the number of rows that satisfy the where clause:

q)exec count i from trade where sym = `aif
6
q)

This is usually what you want.

However, suppose we created another table (perhaps to improve the performance of certain queries) that grouped prices for each symbol:

q)grouped: select price by sym from trade
q)grouped
sym| price                                                   ..
---| --------------------------------------------------------..
acl| 18.1 85.5 86.31 45.65 10.91 31.25 5.49 51.66 1.02 31.82 ..
aif| 44.02 47.74 11.83 14.28 85.11 99.09                     ..
alk| 51.12 16.7 78.71 96.56 26.55 32.09 33.66 30.01 83.24 30...
amg| 59.25 56.18 92.63 46.57 57.25 14.58 69.21 88.25 94.1 28...
aog| 92.59 51.52 96.95 83.2 6.21 59.77 44.19 94.19 3.13 41.94..
..
q)

In table grouped, the price column is of type list-of-float – note the uppercase type letter F:

q)meta grouped
c    | t f a
-----| -----
sym  | s
price| F
q)

It appears that we can retrieve the prices for a given symbol easily enough:

q)exec price from grouped where sym = `aif
44.02 47.74 11.83 14.28 85.11 99.09
q)

Suppose, however, that we want to know how many trades occurred for a particular symbol:

q)exec count price from grouped where sym = `aif
1
q)

What went wrong? A where function yields a list of row indexes that meet the constraints, and then each projection (i.e., the column names between exec and from — in this case, price) yields a list of corresponding field values. Since the number of rows that met our sole constraint is 1, the result of the projection is an untyped list with a single element:

q)type exec price from grouped where sym = `aif
0h
q)count exec price from grouped where sym = `aif
1
q)

Projection results are the arguments to aggregation functions in queries. In other words, the untyped list in our example is the same as the one passed to count. Since the projection’s single element contains the list of prices we want to count, the way out is to combine count with first:

q)exec count first price from grouped where sum = `aif
6
q)

Applying the same logic when counting the trades for every symbol, we need to use count each:

q)select count price by sym from grouped
sym| price
---| -----
acl| 1
aif| 1
alk| 1
amg| 1
aog| 1
..
q)select count each price by sym from grouped
sym| price
---| -----
acl| 14
aif| 6
alk| 10
amg| 12
aog| 13
..
q)

Constraints in where clauses behave the same as well. Consider the following select statement:

q)t
x y z
-------------
a 1 "xyzzy"
b 2 "grue"
c 3 "frobozz"
q)select from t where 5 < count z
x y z
-----
q)

Newcomers to q often expect the above query to return the rows from t whose z column has more than 5 characters (i.e., c 3 frobozz). Rather than counting the contents of each z field, however, count is actually counting the list t `z:

q)count t `z
3
q)t `z
"xyzzy"
"foobar"
"frobozz"
q)

This insight suggests the solution:

q)count each t `z
5 6 7
q)select from t where 5 < count each z
x y z
-------------
c 3 "frobozz"
q)

select sum `long $ size from large_table

In many programming languages, including q, anytime you add integers (unless you somehow know for sure that the sum will fit in – roughly – 31 bits) you risk integer overflow. If you’re lucky, the error will be obvious (e.g., you’ll get a negative number when you expected a positive one). Casting the arguments to sum to long will give you (almost) 63 bits of breathing room.

If your sum won’t fit in 63 bits, you’ll need to explore other options:

• Switch to floats. Although overflow is still possible, it’s rare. However, you lose some precision.
• Use a bignum library such as gmp.
• Use a language (e.g., haskell or clojure) that has built-in support for arbitrary-precision arithmetic.

Lastly, keep in mind that literal values that cannot fit into an int must be suffixed with j – even if the context suggests that a long is expected:

q)select from meta large_table where c = `id
c | t f a
--| -----
id| j
q)count select from large_table where id = 84066472837652480
'84066472837652480
q)count select from large_table where id = 84066472837652480j
1
q)

Although expressions in q are usually evaluated from right to left, there is one exception: the constraints in a where clause are evaluated from left to right. Consider the following example:

q)t: ([] x: `a`b`c; y: 1 2 3)
q)select from t where x < `c, y > 1
x y
---
b 2
q)

The comma separating x < `c from y > 1 is not the join operator; instead, it delimits the constraints of the where clause. If you need to use the join operator in a where clause, use parentheses:

q)select from t where x in (`a, `b), y > 1
x y
---
b 2
q)

Each constraint is evaluated in the usual order, i.e., right-to-left:

q)select from t where x < first -1 # `p`c, y > 1
x y
---
b 2
q)

Because constraints are evaluated from left to right, putting the most restrictive constraint first will speed up queries on large tables. On a date partitioned database, for example, the placement of a constraint on the virtual date column will make a marked performance difference:

select from large_table where date = 2011.02.25, name = `JOE

is significantly faster than

select from large_table where name = `JOE, date = 2011.02.25

The latter query will attempt to load the entire contents of table for rows matching name `JOE prior to narrowing the result set down to a single date. kdb does not provide behind-the-scenes optimization in cases like these.

testfunc: {[]
x: `aaa;
y: `bbb;
z:: `ccc;
-1 "finished!";
}

Let’s demonstrate the placement of a breakpoint prior to the assignment of global variable z. Although there is no explicit support for breakpoints in q, insertion of non-compliant code, such as breakhere; shown below, does the job (don’t forget the trailing semicolon):

testfunc: {[] x: `aaa;
y: `bbb;
breakhere;
z:: `ccc;
-1 “finished!”
}

We are tricking q into throwing a signal ‘breakhere.

q)testfunc[] {[] x: `aaa;
y: `bbb;
breakhere;
z:: `ccc;
-1 “finished!”
}
‘breakhere
q))

At this point, we can examine the local stack.

q))x
`aaa
q))y
`bbb
q))z
()
q))

kdb has suspended execution, leaving the remaining two lines of function testfunc unexecuted. : (colon) resumes execution.

q)):
finished!
-1
q)x
‘x
q)y
‘y
q)z
`ccc
q)

See: global amend (::)

By default, all partitions of a given historical kdb database are available. The .Q.view function enables a given kdb process instance to selectively expose an arbitrary set of partitions into view.

The following example restricts all operations of the database instance to the most recent partition date (typically the previous business day):

q)yesterday: last date
q).Q.view yesterday
q)count date
1
q)yesterday ~ value exec from select distinct date from trade
1b
q)

Here is a more realistic example of a date partitioned historical database where viewable partitions are limited to those year-to-date:

q)start_date: "D" $ string[`year $ .z.D], ".01.01"
q)end_date: last date
q).Q.view date where date within (start_date; end_date)
q)count select from trade where not date within (start_date; end_date)
0
q)

See also: .Q.pn

$ unzip verylargefile.zip /destination_dir/
 I/O error: File too large
$

Make sure your file system can handle LFS.

$ 7za e -o /destination_dir/ verylargefile.zip

See also: 7-zip usage examples.

Many operations are more efficient with a column-oriented approach. In particular, operations that need to access a sequence of values from a particular column are much faster. If all the values in a column have the same size (which is true, by design, in kdb), things get even better. This type of access pattern is typical of the applications for which q and kdb are used.

To make this concrete, let’s examine a column of 64-bit, floating point numbers:

q).Q.w[] `used
108464j
q)t: ([] f: 1000000 ? 1.0)
q).Q.w[] `used
8497328j
q)

As you can see, the memory needed to hold one million 8-byte values is only a little over 8MB. That’s because the data are being stored sequentially in an array. To clarify, let’s create another table:

q)u: update g: 1000000 ? 5.0 from t
q).Q.w[] `used
16885952j
q)

Both t and u are sharing the column f. If q organized its data in rows, the memory usage would have gone up another 8MB. Another way to confirm this is to take a look at k.h.

Now let’s see what happens when we write the table to disk:

q)`:t/ set t
`:t/
q)\ls -l t
"total 15632"
"-rw-r--r--  1 kdbfaq  staff  8000016 May 29 19:57 f"
q)

16 bytes of overhead. Clearly, all of the numbers are being stored sequentially on disk. Efficiency is about avoiding unnecessary work, and here we see that q does exactly what needs to be done when reading and writing a column – no more, no less.

OK, so this approach is space efficient. How does this data layout translate into speed?

If we ask q to sum all 1 million numbers, having the entire list packed tightly together in memory is a tremendous advantage over a row-oriented organization, because we’ll encounter fewer misses at every stage of the memory hierarchy. Avoiding cache misses and page faults is essential to getting performance out of your machine.

Moreover, doing math on a long list of numbers that are all together in memory is a problem that modern CPU instruction sets have special features to handle, including instructions to prefetch array elements that will be needed in the near future. Although those features were originally created to improve PC multimedia performance, they turned out to be great for statistics as well. In addition, the same synergy of locality and CPU features enables column-oriented systems to perform linear searches (e.g., in where clauses on unindexed columns) faster than indexed searches (with their attendant branch prediction failures) up to astonishing row counts.

A thorough introduction to these topics is given in Ulrich Drepper’s ‘What every programmer should know about memory’ and Scott Meyer’s 2011 talk on ‘CPU caches and why you care’.

q)`int $ "A"
65
q)

You can convert a list of characters at once:

q)`int $ "Hello, world!"
72 101 108 108 111 44 32 119 111 114 108 100 33
q)

Alternatively, the left argument to $ (cast) can also be the integer type character, “i” or type value 6h:

q)"i" $ "Hello, world!"
72 101 108 108 111 44 32 119 111 114 108 100 33
q)
q)type 1 2 3
6h
q)6h $ "Hello, world!"
72 101 108 108 111 44 32 119 111 114 108 100 33
q)

See also: Unicode, UTF-8

With the exception of the ? operator, which can be used to provide concurrency protection for sym files, q processes do not coordinate file I/O operations in any way. Thus, kdb splayed table write operations require the same degree of care as ordinary file writes.

The mechanism to protect sym files, however, is used throughout the functions in the .Q namespace that pertain to writing splayed tables. So, you can use those functions to build a database faster by running multiple loader processes in parallel.

Example 1: Concurrent writes to the same table on the same partition. This means two processes are writing to the same (table column) files simultaneously, and that is not safe! The potential for corruption is high.

kdb_proc1: .Q.dpft[`:/vol/db; 2011.04.07; `sym; `t]
kdb_proc2: .Q.dpft[`:/vol/db; 2011.04.07; `sym; `t] 
<span style="color:blue">some *blue* text</span>.

Example 2: Concurrent writes to two different tables on the same partition. This is will not cause corruption.

kdb_proc1: .Q.dpft[`:/vol/db; 2011.04.07; `sym; `t] 
kdb_proc2: .Q.dpft[`:/vol/db; 2011.04.07; `sym; `q] 

Example 3: Concurrent writes to the same table on different partitions. This is also safe, with no potential corruption.

kdb_proc1: .Q.dpft[`:/vol/db; 2011.04.07; `sym; `t]
kdb_proc2: .Q.dpft[`:/vol/db; 2011.04.08; `sym; `t]

If the two partitions are on different I/O channels using segments, this last approach can be much faster than performing the writes serially.

    $ rlwrap q sp.q
    KDB+ 2.7 2011.02.16 Copyright © 1993-2011 Kx Systems
    +`p`city!(`p$`p1`p2`p3`p4`p5`p6`p1`p2;`london`london`london..
    (+(,`color)!,`blue`green`red)!+(,`qty)!,900 1000 1200
    +`s`p`qty!(`s$`s1`s1`s1`s2`s3`s4;`p$`p1`p4`p6`p2`p2`p4;300 ..
    q)sp
    s p qty
    ———
    s1 p1 300
    s1 p2 200
    s1 p3 400
    q)`:splaydir/ set sp
    `:splaydir/
    q)\ls -a splaydir
    ,”.”
    “..”
    “.d” // this is where the col names live
    ,”p”
    “qty”
    ,”s”
    q)get `:splaydir/.d
    `s`p`qty
    q)`:splaydir/.d set reverse get `:splaydir/.d
    `:splaydir/.d
    q)get `:splaydir/.d
    `qty`p`s
    q)

Verify with a fresh kdb instance:

$ rlwrap q splaydir
KDB+ 2.7 2011.02.16 Copyright © 1993-2011 Kx Systems
q)\v
,`splaydir
q)splaydir
qty p s
——-
300 0 0
200 1 0
400 2 0
q)

See also: xcol and xcols faq{.}

If you want to split a string on a delimiter, use the vs (vector from scalar) function:

q)” ” vs “The quick brown fox”
“The”
“quick”
“brown”
“fox”
q)”, ” vs “Hello, world!”
“Hello”
“world!”
q)

When its left argument is the null symbol, `, the vs function breaks apart a symbol on dots:

q)` vs `foo.bar.baz
`foo`bar`baz
q)

Those are the most common cases. We can also split a list into lists of varying length by passing a list of indexes as the left argument to the _ (cut) operator:

q)0 1 4 9 _ til 10
,0
1 2 3
4 5 6 7 8
,9
q)

To split a list of some type other than char using a delimiter is a little more complicated. We start by finding the indexes in the list that match the delimiter:

q)list: 1 2 0 3 4 0 5 6
q)delimiter: 0
q)indexes: where delimiter = list
q)indexes
2 5
q)

Now we can break list into pieces using the _ (cut) operator as above:

q)indexes _ list
0 3 4
0 5 6
q)

This is almost what we want. We’ll use _/: (drop-each-right) to get rid of the delimiters:

q)1 _/: indexes _ list
3 4
5 6
q)

We can grab the first element of the result with # (take):

q)first[indexes] # list
1 2
q)

Then we can just join (using ,) the two together:

q)(enlist first[indexes] # list), 1 _/: indexes _ list
1 2
3 4
5 6
q)

Note that we must call enlist on the front of the list or else we’ll get something a little different from what we intended:

q)(first[indexes] # list), 1 _/: indexes _ list
1
2
3 4
5 6
q)

Lastly, we can generalize to non-atomic types by replacing = with ~/: (match-each-right):

split: {[list; delimiter]
indexes: where delimiter ~/: list;
front: first[indexes] # list;
rest: 1 _/: indexes _ list;
: (enlist front), rest;
}

For example –

q)\l sp.q        / found in $QHOME/sp.q
q)s
s | name  status city
--| -------------------
s1| smith 20     london
s2| jones 10     paris
s3| blake 30     paris
s4| clark 20     london
s5| adams 30     athens
q)meta s
c     | t f a
------| -----
s     | s
name  | s
status| i
city  | s
q)

The output of meta is a kdb table, where columns c and t indicate the column name and data type, respectively.

The key the provides an alternate method:

q)key exec city from s
`symbol
q)

If you need the type code, use the type function:

q)type exec city from s
11h
q)

See also: Type character definitions, Meta output faq{.}.

Unless you use : (return) or ‘ (signal) to specify otherwise, a q function returns the value of its last expression. Since functions are lists of expressions separated by semicolons, both of the following functions return 3:

q)single_expression: {3}
q)single_expression[] 
3
q)last_expression: {1; 2; 3}
q)last_expression[] 
3
q)

To return a value other than that of the last expression, use : (return):

q)explicit_return: {: 3; 4}
q)explicit_return[] 
3
q)

The last expression in explicit_return (i.e., 4) is never reached.

Meanwhile, if a function that does not return via : (return) (or ‘ (signal)) does not have a final expression (i.e, no expression after its last semicolon), that function returns a generic null:

q)no_return: {3; }
q)no_return[] 
q)
q)null no_return[]
1b
q)type no_return[]
101h
q)no_return[] ~ (::)
1b
q)

Lastly, functions can exit in two other ways:

1. Calling the exit function:

q)die: {exit 0}
q)die[] 
$

(By the way, you can add code to run automatically at exit using .z.exit.)

2. Signaling an error:

q)signal: {‘ “oops”}
q)signal[] ‘oops
q)

See this related faq for more information on the use of ‘ (signal)

File handle example:

  q)\touch /tmp/test.txt
  q)key hsym ` $ "/tmp/test.txt"
  `:/tmp/test.txt
  q)

The function key returns an empty list when the file does not exist:

  q)hdel `:/tmp/test.txt
  `:/tmp/test.txt
  q)() ~ key `:/tmp/test.txt
  1b
  q)

key is one of the most heavily overloaded functions in q. It accepts an argument of type dictionary, keyed table, table column, list, and integer. It’s easy to confuse it with keys, which turns out to be an entirely different function.

See also: ~ (match), \ (system), hdel, hsym

q)dict: `a`b`c ! 1 2 3
q)dict 
a| 1 
b| 2 
c| 3
q)`b _ dict
a| 1
c| 3
q)

If you reverse the order of the arguments, the removal still works:

q)dict _ `b
a| 1
c| 3
q)

Although we can’t think of a good reason to do this, you might come across it. As our commenter Attila pointed out, however, you are more likely to see this overload in its assignment form:

q)dict _: `b / same as dict: dict _ `b
q)dict
a| 1
c| 3
q)

Meanwhile, you can remove multiple keys at once by passing a list of keys as the left argument to _ (drop):

q)`a`c _ dict
b| 2
q)

Reversing the arguments does not work in this case:

q)dict _ `a`c
‘type
q)

process_chunk: {[list_of_lines]
// Split and parse each line in list_of_lines and
// then - most likely - create some sort of output.
// After all, the idea here is to avoid holding
// all of the data in memory.
}

and passing your chunk handler along with the file handle to .Q.fs:

    bytes_read: .Q.fs[process_chunk; `:big_file_name] </blockquote> 

For details on how to parse lines of text inside your chunk handler, see this related faq

q)table: ([] x: 5 10 15 20; y: 1 2 3 4)
q)table
x  y
----
5  1
10 2
15 3
20 4
q)select from table where x > 10
x  y
----
15 3
20 4
q)

or SQL via s):

q)s)select * from table where x > 10
x y
—-
15 3
20 4
q)

See also: $QHOME/s.k

1. (types; delimiter) 0: `:filename
2. .Q.fs[chunk_handler; `:filename]

Although there are many ways to read an ASCII file in q – depending on the content, how big the file is, and what you want to do with it – most of the time you will use one of two methods. The first method is for files you want to read into memory in their entirety, while the other approach is for situations in which you want to deal with the file in chunks. The latter scenario is covered in this related faq.

If the file is small enough (compared to the available memory in your system), you can read it all in as a list of lines in one go using read0. Given the following file, lines.txt,

foo=10
bar=20
baz=30

we can write

q)lines: read0 `:lines.txt
q)lines
“foo=10”
“bar=20”
“baz=30”
q)

To break up the lines, we use the vs (vector from scalar) function, applying the /: (each right) adverb so that we split each line:

q)split: “=” vs/: lines
q)split
“foo” “10”
“bar” “20”
“baz” “30”
q)

At this point, you probably want to parse each piece of text into its corresponding type to facilitate fast searching or arithmetic etc. You use the $ (cast) operator to do this, passing an uppercase type character as its left argument:

q)”S” $ “foo”
`foo
q)”I” $ “10”
10
q)

You may remember from this related faq that you can convert a list of items at once:

q)”S” $ (“foo”; “bar”; “baz”)
`foo`bar`baz
q)”I” $ (“10”; “20”; “30”)
10 20 30
q)

But wait! There’s more! If you pass a list of type characters as the left argument to $, you can parse multiple lists:

q)”SI” $ ((“foo”; “bar”; “baz”); (“10”; “20”; “30”))
foo bar baz
10 20 30
q)

The list of type characters can be as long as you like:

q)"SSFI*" $ ("foo"; "bar"; "10.5"; "47"; "left as a string")
`foo
`bar
10.5
47
"left as a string"
q)

Thus, we can parse our file with the following code:

q)”SI” $ flip “=” vs/: read0 `:lines.txt
foo bar baz
10 20 30
q)

Since this sequence of operations is so common, it has been wrapped up in an overload of that workhorse of text I/O, 0: (load text). The trick is to pass a pair as the left argument to 0: where the first element of the pair is the string of type characters and the second element of the pair is the delimiter between each name and value in the file:

q)(“SI”; “=”) 0: `:lines.txt
foo bar baz
10 20 30
q)

Putting it all together, we can turn our file into a table like so:

q)flip `name`val ! (“SI”; “=”) 0: `:lines.txt
name val
——–
foo 10
bar 20
baz 30
q)

Using 0: instead of the combination of read0, vs and $ is faster and less memory-intensive. The differences become significant as the file size grows:

q)system “wc trade_small.csv”
” 1000001 1000001 29997921 trade_small.csv”
q)\ts (“TSIF”; “,”) 0: `:trade_small.csv
554 20971840j
q)\ts “TSIF” $ flip “,” vs/: read0 `:trade_small.csv
2649 232389280j
q)

As a consequence of the 0: function’s superior memory efficiency, it can handle much larger files than the other approach:

q)system “wc trade.csv”
” 10000001 10000001 299888328 trade_big.csv”
q)\ts (“TSIF”; “,”) 0: `:trade_big.csv
5672 335544640j
q)\ts “TSIF” $ flip “,” vs/: read0 `:trade_big.csv
wsfull
q)

If you don’t actually need to parse the file contents, then read0 (by itself) is fine.

By the way, if you only want to grab part of the file, you can pass a triple to read0 in order to read a subset of the bytes. You’ll still get a list of lines broken on newlines:

q)offset: 3
q)number_of_bytes_to_read: 6
q)read0 (`:lines.txt; offset; number_of_bytes_to_read)
“=10”
“ba”
q)

Unless your file has fixed-length records, however, you may find it easier – assuming you have the head and tail utilities (or similar) available, to use thesystem function to get exactly the lines you want. For example,

q)first_line: first system “head -1 lines.txt”
q)

(Note the call to first; system always returns a list of strings, even when there is only one.) This particular example is handy when you need to examine the start of a file to figure out how to read it properly.

q)”I” $ string `123
123
q)

Type number also works:

q)type 123
-6h
q)-6h $ string `123
123
q)

See also: string to symbol conversion faq{.}, Datatypes and $

q)` $ “foo”
`foo
q)

The cast operator also accepts an uppercase type character as its left argument:

q)”S” $ “foo”
`foo
q)

Moreover, you can convert a list of strings at once:

q)”S” $ (“foo”; “bar”; “baz”)
`foo`bar`baz
q)

To go the other way, use the string function:

q)string `foo
“foo”
q)

Since the string function is implemented as the monadic form (i.e., single argument overload) of the k function $ —

q)string
$:
q)

— it is also possible to write the above cast as follows:

q)($)`foo
“foo”
q)

Although we don’t recommend it, we thought you should know in case you see it in the wild.

See also: type

Use the vs (vector from scalar) function:

q)fstring
“f1=va\001f2=vb\001f3=vc”
q)”\001″ vs fstring
“f1=va”
“f2=vb”
“f3=vc”
q)

We can combine vs with /: (each right) to break up each component:

q)”=” vs/: “\001” vs fstring
“f1” “va”
“f2” “vb”
“f3” “vc”
q)

Note that you cannot put any space between vs and /:. If you did, that would be a comment.

We might next turn fstring’s contents into a table using a combination of flip and ! (dict):

q)flip "=" vs/: "\001" vs fstring
"f1" "f2" "f3"
"va" "vb" "vc"
q)`field`value ! flip "=" vs/: "\001" vs fstring
field| "f1" "f2" "f3"
value| "va" "vb" "vc"
q)flip `field`value ! flip "=" vs/: "\001" vs fstring
field value
-----------
"f1"  "va"
"f2"  "vb"
"f3"  "vc"
q)

This is almost what we want. Usually, though, you want the field names to be symbols rather than strings. We can do that by applying (@) the $ (cast) operator to (only) the values of the field column:

q)columns: @[flip "=" vs/: "\001" vs fstring; 0; `$]
q)columns
f1   f2   f3
"va" "vb" "vc"
q)flip `field`value ! columns
field value
-----------
f1    "va"
f2    "vb"
f3    "vc"
q)

However, there is a shortcut we can take to deconstruct the string into a table using one of the many variants of 0::

q)flip `field`value ! “S=\001” 0: fstring
field value
———–
f1 “va”
f2 “vb”
f3 “vc”
q)

Not only is this last example more succinct, it’s much faster:

q)\t do[100000;
flip `field`value !
@[flip "=" vs/: "\001" vs fstring; 0; `$]]
485
q)\t do[100000; flip `field`value ! "S=\001" 0: fstring]
110
q)

See also How do I build a ctrl-A delimited string?

To build delimited strings in general, use the sv (scalar from vector) function:

q)”, ” sv (“Hello”; “world!”)
“Hello, world!”
q)

As in many programming languages, the syntax for non-printable ASCII characters is borrowed from C, although only a subset of C’s sequences is supported:

\n      newline
\r      carriage return
\t      tab

(q also supports a few printable escape sequences: \, &#8221; and &#8217;). However, as in C, you can express any ASCII code in q by following a backslash with the code’s 3 digit octal representation:

q)”\054\040″ sv (“Hello”; “world!”)
“Hello, world!”
q)

Since ctrl-A is ASCII 1, we need to use \001:

q)fstring: “\001” sv (“f1=va”; “f2=vb”; “f3=vc”)
q)fstring
“f1=va\001f2=vb\001f3=vc”
q)

We can use 0: to write fstring to a file, and use an external program to confirm the result:

q)`:test 0: enlist fstring
`:test
q)\\
$ hexdump test
0000000 66 31 3d 76 61 01 66 32 3d 76 62 01 66 33 3d 76
0000010 63 0a
0000012
q)

To break fstring up into its parts, see How do I parse a ctrl-A delimited string?

q)f: {[] do[100000; 2 + 2]}
q)\t f[]
10
q)

\t executes its argument and returns the number of milliseconds that elapsed – i.e., wall clock time (except when you are using \t in its other sense, i.e., setting or getting the time between timer events).

When using kdb as a database server, you will want to use one or more of Simon’s logging scripts as a starting point. These tremendously useful scripts enable you to record every incoming query as well as measure the time taken by every incoming query. That way you can calculate summary statistics on actual database performance in order to find and fix intermittent problems.

q)t: ([] str: ())
q)meta t
c  | t f a
---| -----
str|
q)

A the type of a column defined as () is determined when the first row is inserted:

q)`t insert enlist enlist "foo"
,0
q)select from t
str  
-----
"foo"
q)

See also: Splayed Tables

• k4unit
• qspec

IDEs

• qkdt
• QInsightPad – Public Beta
• studio for kdb+

Libraries

• Perl-Compatible Regular Expressions for q
• q Math Library

Tools

• Doxygen filters
• Linq2Kdb
• kp.net
• perl kx.pm
• Sybase kdb+ driver

See also: Useful companion tools

    q)\c 80 24

You can also use the $LINES and $COLUMNS environment variables to have q start up automatically with your preferred settings:

$ export LINES=40
$ export COLUMNS=120
$ q
KDB+ 2.7 2011.02.16 Copyright © 1993-2011 Kx Systems
q)\c
40 120
q)

To adjust the amount of output when access a q server via a web browser, see http page size (\C)

Lastly, you may be interested in .Q.s and .Q.s1

q).z.ts: {[] show .z.T}
q)\t 500
q)07:41:15.147
07:41:15.647
07:41:16.147
07:41:16.647
07:41:17.147
\t 0
q)

As shown above, \t 0 stops the timer.

To set the timer interval, you must give \t a literal integer. What that means is that, if you want to change the timer frequency at runtime (e.g., when building a simple scheduler), the following will not work:

q)i: 500
q).z.ts: {[] show .z.T; system “t i”; i *: 2}
q)\t 500
q)07:46:47.106
07:46:47.606
07:46:48.106
07:46:48.606
\t 0
q)

Notice what happens when we enter \t i at the repl:

q)\t i
0
q)

What we’ve done is use \t tomeasure the execution time of the expression i. To get the effect we want is, fortunately, easy once you see what is going on:

q)i: 500
q).z.ts: {[] show .z.T; system “t “, string i; i *: 2}
q)\t 500
q)07:59:59.694
08:00:00.195
08:00:01.195
08:00:03.196
\t 0
q)

By passing “t “, string i instead of “t i” to system, the q interpreter sees a literal integer.

One last detail: if you decide you need to delete the definition of .z.ts, you’ll need to use \x.

• $QLIC
• $QHOME
• %WinDir% (only on w32 and w64 OS)
• $QINIT
• $LINES
• $COLUMNS
• $LD_LIBRARY_PATH for dynamic load 2:

You can read the value of any environment variable with the getenv function:

q)getenv `PATH<br /> &#8220;/sw/bin:/sw/sbin:/usr/bin:/bin:/usr/sbin:/sbin:..<br /> q)

Consider Problem 1 from Project Euler: sum the multiples of 3 and 5 which are less than 1000. You may want to try this before reading further. The problems from Project Euler are great exercises for learning q!

The code below shows one solution written in both languages; f is written in q, while g is written in k.

q)f: {(+/) distinct where (0 = x mod 3) | 0 = x mod 5}
q)\
g: {+/?&(0={x-y*x div y}[x;3])|0={x-y*x div y}[x;5]}
\
q)

You can see the bytecode for a function (as well as some generally more useful information, like the function’s arguments) by calling value on it:

q)value f
0xa278a10a020c48a078a10a020c48462531a31b520004
,`x
`symbol$()
,`
3
k){x-y*x div y}
5
+
“{(+/) distinct where (0 = x mod 3) | 0 = x mod 5}”
q)

As you can see, when passed a function, value returns a list. The first element of that list is the function’s bytecode. Now it is easy to show that f and g have the same bytecode:

q)first value f
0xa278a10a020c48a078a10a020c48462531a31b520004
q)first value g
0xa278a10a020c48a078a10a020c48462531a31b520004
q)all (first value f) = first value g
1b
q)

In q, however, you probably would have written the solution a tiny bit differently:

q)f: {sum distinct where (0 = x mod 3) | 0 = x mod 5}
q)

This results in a q version that is actually two bytes smaller and (maybe) a hair faster:

q)first value f
0xa278a10a020c48a078a10a020c48462531390004
q)first value g
0xa278a10a020c48a078a10a020c48462531a31b520004
q)\t do[100000; f til 1000] 3240
q)\t do[100000; g til 1000] 3253
q)

Thanks to Charlie Skelton from kx for pointing out errors in the original version of this article.

q)list: (`sector; 47)
q)type list
0h
q)list ,: 2.7
q)list
`sector
47
2.7
q)

However, you often don’t know what items are going in the list in advance. Let’s try defining an empty list without specifying an element type:

q)list: ()
q)

At first, that seems to be just what we wanted:

q)type list
0h
q)

The trouble begins when we actually add elements to the list:

q)list ,: `sector
q)list ,: 47
‘type
q)

What happened? Let’s re-examine list‘s type:

q)type list
11h
q)

Now list‘s type is no longer 0h; it has been transmuted into 11h; i.e., list of symbol. The ‘type error is telling us that we can’t add an int to a list of symbol.

In some situations, determining the type of an empty list dynamically (i.e., when the first item is added) may be what you want. A list of symbol (or int) uses less memory that an untyped list with the same number of elements. Most operations on a typed list will be faster as well.

In this case, however, we need a way to prevent q from promoting the untyped list to a typed list. The only way to do that is to stuff the list – when you create it – with some data that can’t fit into a typed list. While you could use two values of different types (as in our first example), you only need to use one value if that value is itself a list. Our practice to use a list of characters, i.e., a string:

q)list: enlist “sentinel”
q)type list
0h
q)list ,: `sector
q)list ,: 47
q)count list
3
q)

Note that the use of enlist is essential; without it, list is simply the string:

q)list: (“sentinel”)
q)type list
10h
q)count list
8
q)

We use the string “sentinel” because the term sentinel has been used for decades in computer science to denote a value in a data structure that isn’t actually part of the data, but rather signals some special condition to the program. It also happens to be a term that doesn’t come up often in other domains.

Of course, any code that reads or writes to list needs to understand that the first element is to be ignored. (In our experience, the less code that is aware of that, the less time we spend debugging.) In particular, you must never remove all the data from list; the sentinel must always remain. Otherwise, when you start adding elements again, list‘s type will become list-of-type-of-the-first-element-added:

q)list: list _/ reverse til count list
q)count list
0
q)type list
0h
q)list ,: `sector
q)type list
11h
q)

This can happen unexpectedly when a mixed-type list is actually a column in a table. See this related faq.

f: {[list; n] {[nn; item]
/ n cannot be referenced here,
/ so we pass it as nn
nn + item}[n; ] each list
}

See: Closure

In the example in the question, the / is interpreted as the q adverb over:

q)`ABC/A        / interpreted as `ABC over A.
‘/
q)

The ‘/ error means we have misapplied /. Here’s how to create the symbol we want:

q)`$ “ABC/A”
`ABC/A
q)select from tablename where name = `$ “ABC/A”

Finally, symbols with embedded spaces are allowed, as long they are not leading or trailing:

q)`$” ABC E “
`ABC E
q)string `$” ABC E “
“ABC E”
q)

q)t: ([] x: `a`b; y: 1 2)
q)save `:t.csv
q)\cat t.csv
“x,y”
“a,1”
“b,2”
q)

For other delimiters, you must first format the text using the overload of the 0: function whose first parameter is a character:

q)t: ([] x: `a`b; y: 1 2)
q)”|” 0: t
“x|y”
“a|1”
“b|2”
q)

Notice that, by default, q saves the table column names as a header in the first line of the file. If you want to get rid of the header line, use _ (drop):

q)t: ([] x: `a`b; y: 1 2)
q)1 _ “\t” 0: t
“a\t1”
“b\t2”
q)

Once you have your data as a list of strings complete with delimiters, you use another overload of the 0: function to save the data to disk; this time, the first parameter is the file handle (name) of the destination file:

q)t: ([] x: `a`b; y: 1 2)
q)`:filename.psv 0: 1 _ “|” 0: t
`:filename.psv
q)\cat filename.psv
“a|1”
“b|2”
q)

Since we can’t seem to keep all of the overloads for 0: straight (there are more we didn’t cover here), we like to wrap the above idiom in a function with a descriptive name. For example,

SaveTableDataAsText: {[path; table; delimiter] hsym[path] 0: 1 _ delimiter 0: table}

See also: .h.cd

The exact sequence is as follows:

1. load $QHOME/q.k
2. load $QINIT/q.q or $QHOME/q.q
3. load tables contained in /directorypath
4. load scripts /directorypath/*.q (and *.k or .s) in alphabetical order

$ rlwrap q
KDB+ 2.7 2011.02.16 Copyright (C) 1993-2011 Kx Systems
q).Q.w[]
used| 108432
heap| 67108864
peak| 67108864
wmax| 0
mmap| 0
syms| 537
symw| 15616
q)`1              // force kdb to internalize a new symbol
`1
q).Q.w[]
used| 108432
heap| 67108864
peak| 67108864
wmax| 0
mmap| 0
syms| 538         // increased by 1
symw| 15634       // increase of +18 bytes
q)`12             // force kdb to internalize another
`12
q).Q.w[]
used| 108432
heap| 67108864
peak| 67108864
wmax| 0
mmap| 0
syms| 539         // increase by 1
symw| 15653       // increase of 19 bytes
q)`123
`123
q).Q.w[]
used| 108432
heap| 67108864
peak| 67108864
wmax| 0
mmap| 0
syms| 540         // increase by 1
symw| 15673       // increase of 20 bytes
q)

See this solution to compacting a bloated sym file

q).q.f: {x + y}
q)5 f 6
11
q)

This is generally considered bad style.

select from reallybigtable where date = YYYY.MM.DD

would be a good start.

q)t: ([] thing: ())
q)meta t
c    | t f a
-----| -----
thing|      
q)

True, when you define a table column as an untyped list, it can contain anything. However, the first time you put something in it, the column’s type is set to the type of that first inserted item:

q)`t insert `aunt
,0
q)`t insert 33.33
'type
  [0]  `t insert 33.33
          ^
q)meta t
c    | t f a
-----| -----
thing| s    
q)

In our example, the type of the thing column has been promoted to symbol; we can’t insert numbers into it.

When you want to store elements of different types in the same column, you have to prevent q from performing this type promotion by stuffing a sentinel row (i.e., a row of dummy values), whose value for the column in question is itself a list, into the table:

q)t:([] thing: enlist "sentinel")
q)meta t
c    | t f a
-----| -----
thing| C    
q)

Don’t believe the hype. The meta function is trying to help out by actually inspecting the first value in the column when reporting the thing column’s type. However, the true type of the column is untyped, i.e., zero:

q)type t `thing
0h
q)

You can, therefore, put whatever you like into t’s thing column:

q)t: ([] thing: enlist "sentinel")
q)meta t
c    | t f a
-----| -----
thing| C    
q)q)`t insert/: (`gave; 42)
1
2
q)t
thing     
----------
"sentinel"
`gave     
42        
q)meta t
c    | t f a
-----| -----
thing| C    
q)

Beware cleaning out such a table too thoroughly (in, say, an end-of-day function):

q)delete from `t
`t
q)meta t
c    | t f a
-----| -----
thing|      
q)

See this related faq on untyped lists.

• you have developers who know how to program them
• you don’t mind putting your data on a cloud
• the data you need for even a single query can’t fit in memory
• you can afford to wait minutes for your query results

For some applications, then, clouds are an excellent approach. kdb+, however, is typically used in scenarios where

• you can’t (e.g., due to licensing) or don’t want your data outside your firm
• you don’t have an in-house cloud infrastructure
• you don’t want each query to be an IT project in and of itself
• you want the answer to your query in milliseconds (or seconds at worst)
• you can fit your data in memory (not necessarily all at once)

q)now: .z.T
q)now
00:15:00.812
q)now mod 1000
00:00:00.812
q)

If you want the milliseconds as an integer, you can simply follow the mod with a conversion to int:

q)`int$ now mod 1000
812
q)

You don’t want to pass a datetime to mod; the result you’ll get is not what you had in mind. You must convert it to a time first:

q)now: .z.Z
q)now
2011.03.26T09:51:26.624
q)now mod 1000
102.4107
q)`int$ (`time$ now) mod 1000
624
q)

$ rlwrap q
KDB+ 2.7 2011.02.16 Copyright © 1993-2011 Kx Systems
q).Q.gc[]
0j                            / as expected, gc is a no-op
q).Q.w[] 
used| 108432
heap| 67108864
peak| 67108864
wmax| 0
mmap| 0
syms| 538
symw| 15638
q)-1000000?`6                 / 1 million symbols of length 6
`milgli`igfbag`kaodhb`bafclb`kfhogj`jecpae`kfmohp`lk..
q).Q.w[] used
used| 108592
heap| 67108864
peak| 67108864
wmax| 0
mmap| 0
syms| 1000539                 / bumped by a million
symw| 31400168
q).Q.gc[]
0j                            / no-op again!
q).Q.w[]
used| 108592
heap| 67108864
peak| 67108864
wmax| 0
mmap| 0
syms| 1000539                 / remains unchanged
symw| 31400168
q)

If you do need to distinguish different operating systems, use the variable .z.o to detect the operating system in which your code is running.

q)x: 1 2 3
q)x 3
0N
q)

Thus, we can take the item whose corresponding null we want, make a list from it, and then access that list with an out-of-range index:

q)NullOf: {[item] enlist[item] 1}
q)NullOf 1
0N
q)(::) ~ NullOf {x*x}
1b
q)null NullOf (.z.D; 1e; ” “; type “foo”)
1111b
q)

Remember, there is no null list. If you pass a list to NullOf, you get the empty list of the corresponding type:

q)NullOf 1 2 3
`int$()
q)

See also: .Q.ty

q)flip `a`b`c ! (1 2 3; 1 2 3; 1 2 3)
a b c
-----
1 1 1
2 2 2
3 3 3
q)1! flip `a`b`c ! (1 2 3; 1 2 3; 1 2 3)
a| b c
-| ---
1| 1 1
2| 2 2
3| 3 3
q)2! flip `a`b`c ! (1 2 3; 1 2 3; 1 2 3)
a b| c
---| -
1 1| 1
2 2| 2
3 3| 3
q)

When !’s left argument is zero, it returns an unkeyed table from a keyed table:

q)0! flip `a`b`c ! (1 2 3; 1 2 3; 1 2 3)
a b c
-----
1 1 1
2 2 2
3 3 3
q)

An arguably more readable equivalent is the xkey function:

q)`a xkey flip `a`b`c ! (1 2 3; 1 2 3; 1 2 3)
a| b c
-| ---
1| 1 1
2| 2 2
3| 3 3
q)`a`b xkey flip `a`b`c ! (1 2 3; 1 2 3; 1 2 3)
a b| c
---| -
1 1| 1
2 2| 2
3 3| 3
q)() xkey flip `a`b`c ! (1 2 3; 1 2 3; 1 2 3)
a b c
-----
1 1 1
2 2 2
3 3 3
q)

It’s easy to confuse function xkey with key, keys, xcol and xcols.

one can verify the read-only mode by examining the return value of function _

$ q -b
KDB+ 2.7 2011.02.16 Copyright (C) 1993-2011 Kx Systems
...
q)\_
1
q)

1. The index column, i, which is present in all tables, and
2. the partition column on a partitioned table. For example, the most common partitioning scheme is by date. Rather than storing the date column along with the other columns in the table, the date is inferred from the partition directory; splayed table directories will not contain files for the date column.

See also: .Q.pf and .Q.pt

$ rlwrap q sp.q
KDB+ 2.7 2011.02.16 Copyright © 1993-2011 Kx Systems
+`p`city!(`p$`p1`p2`p3`p4`p5`p6`p1`p2;`london`london`london`l..
(+(,`color)!,`blue`green`red)!+(,`qty)!,900 1000 1200
+`s`p`qty!(`s$`s1`s1`s1`s2`s3`s4;`p$`p1`p4`p6`p2`p2`p4;300 200 100 400 200 300)
q)sp
s p qty
———
s1 p1 300
s1 p2 200
s1 p3 400
s1 p4 200
q)`p`s xcol sp // rename the first two cols
p s qty
———
s1 p1 300
s1 p2 200
s1 p3 400
s1 p4 200
q)

Use xcols to reorder columns:

q)(reverse cols sp) xcols sp
qty p s
———
300 p1 s1
200 p2 s1
400 p3 s1
200 p4 s1
q)

One way to remember which is which is that, just as rename precedes reorder (in alphabetical order), so does xcol precede xcols.

See this solution to compacting a bloated sym file.

see also: -W, .z.D, mod, os-commands

$ rlwrap q sp.q
KDB+ 2.7 2011.02.16 Copyright © 1993-2011 Kx Systems
q)type meta sp
99h
q)meta sp
c  | t f a
---| -----
s  | s s
p  | s p
qty| i
q)cols meta sp
`c`t`f`a
q)meta meta sp
c| t f a
-| -----
c| s
t| c
f| s
a| s
q)

For example, cell carriers use time series databases to monitor peak mobile phone usage. Similarly, utilities use them to study optimal tariff structure to shape on-peak or off-peak electricity usage.

See wikipedia for more on time series and column-oriented databases.

    delete entity from namespace

e.g.

    delete foo from `.

However, .z is a special namespace, so delete doesn’t work:

q).z.ts: {47}
q)delete ts from `.z
‘type
q)

There is a special command just for removing elements from .z:

q)\x .z.pi

See \x (expunge).

Use symbol when the set of values is restricted, e.g., exchange names, or tickers. Symbols are faster for comparison operations (and that means lookup as well).

See string interning.

xgroup.

Explanation

When you write a grouping query (i.e., one with a by clause) that lists columns in its select clause, the result nests all of the values from the selected columns. However, if no columns are listed in the select clause, the result only includes each column’s last value for each key.

For example, consider the following table:

trade: ([] time:  `time$   ();
           sym:   `symbol$ ();
           price: `float$  ();
           size:  `int$    ());
`trade insert (09:30:00t + til 1000; 
	1000 # 10 ? `3; 
	10.0 + 1000 ? 15.0; 
	100 + 100 * 1000 ? 10);

If we execute a query with a column in its select clause, the result is nested:

q)select price by sym from trade
sym| price                                         ..
---| ----------------------------------------------..
dgh| 13.68204 18.47666 19.59256 23.92803 12.75742 2..
dmo| 19.39943 19.1779  12.71798 10.9528  24.89916 2..
ekh| 22.16764 19.13283 23.42802 11.1409  24.42824 1..
gap| 16.33531 11.47509 13.90435 16.40958 22.59581 2..
log| 23.23925 17.83933 21.00673 14.97167 18.14587 1..
q)

However, if the select clause is empty, then the query returns only the last item from each group:

q)select by sym from trade
sym| time         price    size
---| --------------------------
dgh| 09:30:00.992 22.55807 1000
dmo| 09:30:00.999 22.72169 1000
ekh| 09:30:00.993 24.27423 300
gap| 09:30:00.995 20.08908 900
log| 09:30:00.998 24.42545 600
q)

Usually, this behavior is what you want. However, if you want to group all columns (except the grouping column), you have two options:

1. List each column explicitly in your select clause,

q)select time, price, size by sym from trade
sym| time                                          ..
---| ----------------------------------------------..
dgh| 09:30:00.002 09:30:00.012 09:30:00.022 09:30:0..
dmo| 09:30:00.009 09:30:00.019 09:30:00.029 09:30:0..
ekh| 09:30:00.003 09:30:00.013 09:30:00.023 09:30:0..
gap| 09:30:00.005 09:30:00.015 09:30:00.025 09:30:0..
log| 09:30:00.008 09:30:00.018 09:30:00.028 09:30:0..
q)

or,

2. Use xgroup:

q)`sym xgroup trade
sym| time                                          ..
---| ----------------------------------------------..
pln| 09:30:00.000 09:30:00.010 09:30:00.020 09:30:0..
nch| 09:30:00.001 09:30:00.011 09:30:00.021 09:30:0..
dgh| 09:30:00.002 09:30:00.012 09:30:00.022 09:30:0..
ekh| 09:30:00.003 09:30:00.013 09:30:00.023 09:30:0..
ojn| 09:30:00.004 09:30:00.014 09:30:00.024 09:30:0..
q)

The more columns your table has, the more attractive xgroup will be. However, every time we tried, xgroup was slightly slower:

q)\t do[10000; select time, price, size by sym from trade]
246
q)\t do[10000; `sym xgroup trade]
275
q)

On the other hand, if you want the code in question to be generically applied to several tables, xgroup is much better than the alternatives.

Notice that xgroup, in contrast to select by, does not sort the result by its key. If you need the rows sorted by key, follow xgroup with xasc:

q)`sym xasc `sym xgroup trade
sym| time                                          ..
---| ----------------------------------------------..
dgh| 09:30:00.002 09:30:00.012 09:30:00.022 09:30:0..
dmo| 09:30:00.009 09:30:00.019 09:30:00.029 09:30:0..
ekh| 09:30:00.003 09:30:00.013 09:30:00.023 09:30:0..
gap| 09:30:00.005 09:30:00.015 09:30:00.025 09:30:0..
log| 09:30:00.008 09:30:00.018 09:30:00.028 09:30:0..
q)

The cost of the sort will make the xgroup significantly slower than using select:

q)\t do[10000; `sym xasc `sym xgroup trade]
479
q)

q)2 5 # til 10
0 1 2 3 4
5 6 7 8 9
q)

When # is used in this way, it is referred to as “reshape.” We like to call this, “taking a box.” Just like when you apply # with an integer left argument, taking a box stops when you have taken as many items as requested, leaving the rest behind:

q)3 3 # til 10
0 1 2
3 4 5
6 7 8
q)

Similarly, when you over-take a box, you start over from the beginning of the source:

q)4 3 # til 10
0 1 2
3 4 5
6 7 8
9 0 1
q)

If you want to take all the items from the source, pass 0N as the first element of the box:

q)0N 2 # til 10
0 1
2 3
4 5
6 7
8 9
q)

If you take a whole box, but the elements from the source don’t fit, you end up with a short list in the last row:

q)0N 3 # til 10
0 1 2
3 4 5
6 7 8
,9
q)

See also: cut and faq on _ (drop)

    http://server:5001/.csv?select from tablename where date=2011.02.25,fname=`JOE

See also: .h.cd

q)d: (-10000 ? `4) ¡ til 10000
q)key d
`namk`eefb`dnkg`pfme`ndpk`bhck`bdga`hkdk`gpja`oiof..
q)value d
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ..
q)\t do[100000; d key[d] 5000] 443
q)d: (asc key d) ! value[d] iasc key d
q)key d
`s#`aaab`aaak`aaap`aabb`aabn`aace`aadj`aaeg`aaeh`a..
q)\t do[100000; d key[d] 5000] 89
q)

Notice that value d above did not have the `s attribute; that is, the til function does not return a list with the `s attribute applied. In general, for a list to have the `s attribute, you must either sort the list (with asc as above) or let q know explicitly that the list is sorted by applying the `s attribute to a list that you know is already sorted:

q)x: `s # til 10
q)x
`s#0 1 2 3 4 5 6 7 8 9
q)

If you try to apply `s to a list that isn’t actually sorted, you’ll get an ‘s-fail error:

q)`s # 3 2 1
‘s-fail
q)

You can add elements to the end of a sorted list without losing the sorted attribute if the new elements do not violate the sorted-ness of the result:

q)x: `s # 1 2 3
q)x
`s#1 2 3
q)x ,: 4 5 6
q)x
`s#1 2 3 4 5 6
q)x ,: 7 7 7
q)x
`s#1 2 3 4 5 6 7 7 7
q)

However, other modifications to a sorted list will cause the list to lose its attribute:

q)x: `s # 1 2 3
q)x
`s#1 2 3
q)x, 0
1 2 3 0
q)reverse x / reverse function removes it
3 2 1
q)

This is true even if the modifications result in a sorted list:

q) x: `s # 1 2 3
q)1 _ x
2 3
q)2 # x
1 2
q)

You can remove the sorted attribute explicitly by applying a null attribute to the list:

q)`#1 2 3 / application of a null attribute
1 2 3
q)

See also the following function definitions: attr and #.

For even more detail, see Section 42 of the Abridged Q Language Manual.

This command allows kdb to continue running and servicing requests while a new partition is constructed. Running kdb remains unware of the new partition until the \l . command is issued.

See also: .Q.l

1. at the start of a non-empty line,

/ this line is ignored
/so is this one

or

2. is preceded by a space:

foo: 47;      / this is also a comment
42+/1 2 3     / The first / does NOT start a comment!

In the last example, the first slash functions as the adverb over.

Begin a block comment with a lone backslash at the start of a line, and finish it with sole slash –

\
All of this is a block comment.
Even /\ this is OK.
/
x: 47              / this line (before the slash) is code

or the reverse:

/
The opposite works, too.
Using block comments to comment out code can be
confusing unless you pick one way and stick to it.
\

A \ does not have to be matched; it then serves as an end-of-file indicator:

\
all lines below are
i
g
n
o
r
e
d

Lastly, if a script’s first line begins with ‘#!’, that line is skipped, e.g.:

#! blah.. blah..

Inside a function, the double backslash requires escaping. Use value "\\\\" or system \\

Note: kdb process exits can be trapped via .z.exit.

$ rlwrap q
KDB+ 2.7 2011.02.16 Copyright © 1993-2011 Kx Systems
q).Q.w[] 
used| 108432   / bytes malloced
heap| 67108864 / heap bytes available
peak| 67108864 / heap high-watermark in bytes
wmax| 0        / workspace limit from -w param
mmap| 0        / amount of memory mapped
syms| 537      / number of symbols interned
symw| 15616    / bytes used by 537 symbols
q)

Note how .Q.w is simply a pretty print output of \w and the workspace memory footprint from \w 0. 

q).Q.w
k){`used`heap`peak`wmax`mmap`syms`symw!(.”\\w”),.”\\w 0″}
q)\w
108304 67108864 67108864 0 0j
q)

See also: view, views, :: , .z.b and \b

select [10] from table where date=2011.02.27
select [-10] from table where date=2011.02.27

If the table is in memory, you can use the virtual column i to retrieve a specific range of rows:

If you want to retrieve specific rows by index from a partitioned table, use .Q.ind.

C.f. # (take) and sublist

If you want to call the file something else, or keep it separate from your q installation, you can set the environment variable QINIT to the path to your initialization file.

It is also possible to add k code to q.k (which will be executed before q.q), although we’ve never needed to.

See also: kdb error code table

(“SSS”; enlist “,”) 0: `:filename.csv

See also: field parsing and operator 0:

1. memory
2. disk i/o
3. cpu

q)f: {x + y + z}
q)f[1; 2; 3] 6
q){x + y + z}[1; 2; 3] 6
q)

If z appears in the definition of an argument-less function, then the function takes 3 arguments. It is not necessary for x or y to appear:

q)f: {z}
q)f[3]
{z}[3]
q)f[3; 4; 5]
5
q)

Similarly, if y appears but z does not appear, then the function takes 2 arguments. What may surprise you is that, if none of x, y, or z appear, then the function takes a single argument called x:

q)f: {42}
q)f[] 42
q)value f
0xa00001
,`x
`symbol$()
,`
42
“{42}”
q)

This is only of practical import if you want to test a function for how many arguments it takes. If you really want to define a function taking no arguments, you might try to use an empty parameter list:

q)f: {[] 47}
q)f[] 42
q)value f
0xa00001
,`
`symbol$()
,`
47
“{[] 47}”
q)

While we’ve succeeded only in defining a function with a single, unnamed parameter, all is not lost. If you follow a convention of always declaring nullary functions in this way, that’s enough to test at runtime how many arguments a function requires.

Although it can’t directly tell you the name of the function in which you’ve trapped, it does know the body of the current function via the variable .z.s (self). Using .z.s (self), you can use the following handy code to deduce the name of the function:

function_name: {[body] names: value “\\f”;
bodies: value each names;
matches: names where body ~/: bodies;
$[0 = count matches;
‘ “Not found (try a different namespace?)”;
  1 = count matches;
first matches;
/ else
matches]}

Here’s how you use it:

q)f: {break}
q)f[] {break}
‘break
q))function_name .z.s
`f
q))

Things get more complicated if you put functions into different namespaces:

q)\d .foo
q.foo)bar: {break}
q.foo)\d .
q).foo.bar[] {break}
‘break
q.foo))`.[`function_name] .z.s
`bar
q.foo))

As you can see, q puts you into the namespace that was in effect at the time the function was defined. You may want to put function_name into a file that you load into every namespace so you don’t have to remember to type `.[`function_name].

Note that that namespace that q breaks into is not necessarily the namespace in which the function is defined:

q).foo.baz: {break}
q).foo.baz[] {break;}
‘break
q))function_name .z.s
‘Not found (try a different namespace?)
q))\d .foo
q.foo))`.[`function_name] .z.s
`baz
q.foo))

Especially interesting, regarding this last example, is that function_name returns the correct function even though bar and baz have identical bodies:

q.foo))bar
{break}
q.foo)baz
{break}
q.foo))string[bar] ~ string baz
1b
q.foo))bar ~ baz
0b
q.foo))

• rlwrap
• rlwrap (Mac OS X)
• taskset

Useful

• screen
• byobu

See also: kdb related software

In any exception handling model, there is a block of code to attempt to execute and one or more blocks of code to invoke should the attempted block fail. Moreover, the error handling blocks have a proscribed form. Lastly, there is a mechanism to raise an exception.

Let’s look at a simple example in q and break it down:

LaidBackLoad: {[file]
  @[{system "l ", x; 1b}; / code to attempt
    file;                 / argument
    {[err] 0N! err; 0b}]} / error handler

In q, there is only one error handling block, and both the try block and the error handling block must be functions. In addition, the error handler takes a single parameter, which – if the error handler is called – will be a string.

In our example, the function to try takes only a single parameter, and that’s why the protected execution expression starts with an @. If your try function takes more than one parameter, use . instead:

LaidBackEqual: {
  .[{x = y};              / code to attempt
    (x; y);               / arguments
    {[err] 0N! err; 0b}]} / error handler

You signal an error (which may be a symbol or a string) using the ‘ (signal) operator:

CantBeNegative: {if [x < 0; ‘ “Must be >= 0”]; x}

This is exactly the same mechanism that q uses to let us know when we have done something wrong:

q)(1 2 3) = 1 2
‘length
q)

There is one last detail to note when reviewing the above examples. Protected execution expressions are expressions, and they have a value. If the try function does not signal, then the expression’s value is the try function’s return value. Otherwise, the protected execution expression’s value is the value returned by the error handler.

$ q sp.q
KDB+ 2.7 2011.02.16 Copyright © 1993-2011 Kx Systems
+`p`city!(`p$`p1`p2`p3`p4`p5`p6`p1`p2;`london`london`london`l..
(+(,`color)!,`blue`green`red)!+(,`qty)!,900 1000 1200
+`s`p`qty!(`s$`s1`s1`s1`s2`s3`s4;`p$`p1`p4`p6`p2`p2`p4;300 200 100 400 200 300)
q)\a
`p`s`sp
q)\a .
`p`s`sp
q)tables `.
`p`s`sp
q)

See wikipedia’s definition.

delete xyz from `t // in place delete (backquote)
delete xyz from t  // returns a copy minus column xyz

The latter rule applies, for example, to functions. The following is not valid q:

Even a function’s closing brace must be indented if it appears on a line by itself:

Function definition is not the only time you have to be careful to follow this rule, however. In the following example, g is not the list (47; 48); rather, g is a projection of the comma operator with its first argument set to 47:

g: 47,
48

James Gosling stated that he purposely made Java syntax C-like to facilitate the adoption of the JVM platform. q’s query language was similarly designed to resemble SQL. Begin with what you know, and incrementally refine your queries using q-specific features to get the best performance possible.

    delete from `t where i = n

Don’t forget the backquote if you intend to delete-in-place.

• pass them as arguments to other functions
• return them from functions
• create new functions at run time

In fact, since the syntax of q is consistent with the semantic similarity between calling a function and indexing a data structure, sometimes you won’t even care whether a particular value is a function or data.

Tables can be treated as ordinary data as well. In a typical database, tables can only be manipulated and queried using a special language via a specific access mechanism (.NET’s LINQ is Microsoft’s effort to bridge the gap). In q, although you can use a query language when convenient, you can also use any other feature of the q language that you might want. For example, sometime it’s much simpler or faster to express a table update using . (apply).

See also: error codes

See $QHOME/q.k to see how much of q is written in k.

See also: kdb error code table

However, with memory mapped tables, the size of your table can theoretically be as large as your disk (assuming it does not exceed the kdb limitations{.}). As long as the operations you perform on your database don’t require more than your RAM can handle at once, you’ll be fine. That’s how kdb has been handling multi-terabyte databases for years.

If your goal is to prevent clients from being able to read your code, see this related faq.

Draw: {[shape] shape[`draw] shape}

(Sadly, we cannot write shape.draw shape; dot notation is not supported for local variables, including arguments.)

Given an appropriate set of constructors –

NewCircle: {[x; y; radius] `x`y`radius`draw ! (x; y; radius; {[circle] … })}
NewRectangle: {[x; y; w; h] `x`y`w`h`draw ! (x; y; w; h; {[rect] …})}

etc, the following does exactly what you would expect:

shapes: (NewCircle[0; 0; 1];
NewRectangle[-1; -2; 4; 2]);
Draw each shapes;

q)t: ([] x: `a`b; y: 1 2)
q)u: ([] z: "YN")
q)t ^ u / works only when count[t] = count[u] 
x y z
-----
a 1 Y
b 2 N
q)

q)asc 3 5 2 0 1
`s#0 1 2 3 5
q)desc 3 5 2 0 1
5 3 2 1 0
q)

Notice that, in the case of asc, q attaches the `s attribute to the result. (There is no corresponding attribute for a descending sort.)

As an aside, you may be surprised to discover that calling asc on an attribute-free list that happens to be in sorted order will have the `s attribute applied to it as a side effect:

q)list: til 1000
q)attr list
`
q)asc list
`s#0 1 2 3 4 5 6 7 8 9 10 ..
q)list
`s#0 1 2 3 4 5 6 7 8 9 10 ..
q)

Also take a look at iasc and idesc.

Finally, to sort a table instead of a list, see xasc and xdesc. Read more about sorting tables from this faq

For example:

$ cat /vol/db/par.txt
/vol/disk1/segment
/vol/disk2/segment
/vol/disk3/segment
$

$ ls /vol/disk*/segment/2011.03.07/vltable 
/vol/disk1/segment/2011.03.07/vltable # sym A-I
/vol/disk2/segment/2011.03.07/vltable # sym J-R
/vol/disk3/segment/2011.03.07/vltable # sym R-Z
$

Note that the typical `p# attribute of column sym will be missing in the query result. Remember to re-apply the attribute prior to use.

See also: .Q.u, and .Q.par

q)1 + 0N! til 10
0 1 2 3 4 5 6 7 8 9
1 2 3 4 5 6 7 8 9 10
q)

0N! does change the functionality of your code in one subtle, albeit usually harmless, way. If you place 0N! before a modifier-assignment operator (e.g. +:, -:), then the result of that assignment is no longer null; it is the value assigned:

q){x +: 47} 1
q){0N! x +: 47} 1
48
48
q)

See also: .Q.s and .Q.s1

$ q -e 1

or from the q console

    q)\e 1

Upon an error, kdb halts and outputs the body of the function (.z.s (self)) in which the error occurred as well as an error message. You are free to inspect the values of any global or local variables to try to diagnose the source of the problem. At this point you have the following options:

• type ‘(single quote) to pop the stack.
• type :(colon) to resume execution
• type \(slash) to exit debug mode

There is no ability step into a function call or move up and down the stack.

• lists
• dictionaries
• tables

However, q does use other structures internally (e.g., hash tables) to enhance the performance of operations on the above language-level structures.

What do you do if you want to have the callee modify variables passed by the caller? As our reader cyprus points out, you can pass the name of a global variable:

modifier: {[variable_name]
 variable_name set 47;
 }
caller: {[]
 global:: "global";
 modifier `global;
 show global; // prints 47
 }

Whether this should be called pass-by-reference is the subject of many flame wars; Wikipedia’s entry on Evaluation strategy provides a cogent, balanced explanation of the varying interpretations of that (and related) terms. What’s important to know are the following:

• it is impossible for a callee to modify a caller’s local variables, and
• passing a large structure between functions is inexpensive until a callee modifies it.