Monday, October 30, 2006

vi and csope


The Vim/Cscope tutorial

Cscope is a very handy tool, but it's even better when you don't ever have to leave the comfort of your favorite editor (i.e. Vim) to use it. Fortunately, Cscope support has been built into Vim.

This tutorial introduces you both to Vim's built-in Cscope support, and to a set of maps that make searching more convenient.

It is assumed you know the basics of using a vi-style editor, but you don't need any particular knowledge about Vim (where Vim-specific features--like multiple windows--are used, a working knowledge of the features is briefly introduced). You also don't need to know anything about Cscope: the basics are introduced as we go along.

In a nutshell, Vim's Cscope support is very similar to Vim's ctags features, in case you've used those. But since Cscope has more search types than ctags, there are a few differences.

This is a hands-on tutorial, so open up a shell, and follow these steps:

1. Get and install Cscope if you don't have it already on your machine. Ideally, you will also have Vim 6.x, but you can get most of the functionality with later versions of Vim 5 (vertical splits don't work, but horizontal splits will work if you modify the maps as described in the file's comments).

Note: If your version of Vim wasn't compiled with '--enable-cscope', you will need to reconfigure and recompile Vim with that flag. Most Vim binaries that ship with Linux distributions have the Cscope plugin enabled.

2. Download the cscope_maps.vim file, and arrange for it to be read by Vim at startup time. If you are using Vim 6.x, stick the file in your $HOME/.vim/plugin directory (or in any other 'plugin' subdirectory in your 'runtimepath'). If you are using Vim 5.x, you can either cut and paste the entire contents of the cscope_maps file into your $HOME/.vimrc file, or stick a "source cscope_maps.vim" line into your .vimrc file.

3. Go into a directory with some C code in it, and enter 'cscope -R' (the '-R' makes Cscope parse all subdirectories, not just the current directory). Since we aren't passing the '-b' flag (which tells Cscope to just build the database, then exit), you will also find yourself inside Cscope's curses-based GUI. Try a couple of searches (hint: you use the arrow keys to move around between search types, and 'tab' to switch between the search types and your search results). Hit the number at the far left of a search result, and Cscope will open Vim right to that location. (unless you've set your EDITOR environment variable to something besides Vim). Exit Vim, and you'll be right back in the Cscope GUI where you left off. Nifty.

Alas, the Cscope interface has one big problem: you need to exit Vim each time you want to do a new search. That's where the Vim plugin comes in. Hit CTRL-D to exit Cscope.

4. Start up Vim. If you want, you can start it with a C symbol (ex: 'vim -t main'), and you should hop right to the definition of that symbol in your code.

5. Put the cursor over a C symbol that is used in several places in your program. Type "CTRL-\ s" (Control-backslash, then just 's') in quick succession, and you should see a menu at the bottom of your Vim window showing you all the uses of the symbol in the program. Select one of them and hit enter, and you'll jump to that use. As with ctags, you can hit "CTRL-t" to jump back to your original location before the search (and you can nest searches and CTRL-t will unwind them one at a time).

Mnemonic: the '\' key is right next to the ']' key, which is used for ctags searches.

6. Try the same search, but this time via "CTRL-spacebar s". This time, your Vim window will split in two horizontally , and the Cscope search result will be put in the new window. [if you've never used multiple Vim windows before: move between windows via 'CTRL-W w' (or CTRL-W arrow key, or CTRL-W h/j/k/l for left/up/down/right), close a window via 'CTRL-W c' (or good old ':q'), make the current window the only one via 'CTRL-W o', split a window into two via 'CTRL-W s' (or 'CTRL-W v' for a vertical split), open a file in a new window via ':spl[it] filename']

Mnemonic: there's now a big, spacebar-like bar across the middle of your screen separating your Vim windows.

7. Now try the same search via "CTRL-spacebar CTRL-spacebar s" (just hold down the CTRL key and tap the spacebar twice). If you have trouble hitting the keys fast enough for this to work, go into the cscope_maps.vim script and change Vim's timeout settings as described in the comments [actually, I generally recommend that you turn off Vim's timeouts]. This time your Vim window will be split vertically (note: this doesn't work with Vim 5.x, as vertical splits are new with Vim 6.0).

8. Up to now we've only been using the keystroke maps from 'cscope_maps.vim', which all do a search for the term that happens to be under your cursor in Vim. To do Cscope searches the old-fashioned way (using Vim's built-in Cscope support), enter ":cscope find symbol foo" (or, more tersely, ":cs f s foo"). To do the horizontal split version, use ":scscope" (or just ":scs") instead (Vim 6.x only). While it's easier to use the maps if the word you want to search for is under your cursor, the command line interface lets you go to any symbol you type in, so you'll definitely want to use it at times.

9. So far we've only been doing one kind of search: 's', for 'find all uses of symbol X'. Try doing one of Cscope's other searches by using a different letter: 'g' finds the global definition(s) of a symbol, 'c' finds all calls to a function, 'f' opens the filename under the cursor (note: since Cscope by default parses all C header files it finds in /usr/include, you can open up most standard include files with this). Those are the ones I use most frequently, but there are others (look in the cscope_maps.vim file for all of them, and/or read the Cscope man page).

10. Although Cscope was originally intended only for use with C code, it's actually a very flexible tool that works well with languages like C++ and Java. You can think of it as a generic 'grep' database, with the ability to recognize certain additional constructs like function calls and variable definitions. By default Cscope only parses C, lex, and yacc files (.c, .h, .l, .y) in the current directory (and subdirectories, if you pass the -R flag), and there's currently no way to change that list of file extensions (yes, we ought to change that). So instead you have to make a list of the files that you want to parse, and call it 'cscope.files' (you can call it anything you want if you invoke 'cscope -i foofile'). An easy (and very flexible) way to do this is via the trusty Unix 'find' command:

find . -name '*.java' > cscope.files

Now run 'cscope -b' to rebuild the database (the -b just builds the database without launching the Cscope GUI), and you'll be able to browse all the symbols in your Java files. Apparently there are folks out there using Cscope to browse and edit large volumes of documentation files, which shows how flexible Cscope's parser is.

For larger projects, you may additionally need to use the -q flag, and/or use a more sophisticated 'find' command. See our tutorial on using Cscope with large projects for more info.

11. Try setting the $CSCOPE_DB environment variable to point to a Cscope database you create, so you won't always need to launch Vim in the same directory as the database. This is particularly useful for projects where code is split into multiple subdirectories. Note: for this to work, you should build the database with absolute pathnames: cd to /, and do

find /my/project/dir -name '*.c' -o -name '*.h' > /foo/cscope.files

Then run Cscope in the same directory as the cscope.files file (or use 'cscope -i /foo/cscope.files'), then set and export the $CSCOPE_DB variable, pointing it to the cscope.out file that results):

cd /foo
cscope -b
CSCOPE_DB=/foo/cscope.out; export CSCOPE_DB

(The last command above is for Bourne/Korn/Bash shells: I've forgotten how to export variables in csh-based shells, since I avoid them like the plague).

You should now be able to run 'vim -t foo' in any directory on your machine and have Vim jump right to the definition of 'foo'. I tend to write little shell scripts (that just define and export CSCOPE_DB) for all my different projects, which lets me switch between them with a simple 'source projectA' command.
BUG: in versions of Cscope prior to 15.4, there is a silly bug that may cause Vim to freeze when you do this unless you call your database something other than the default 'cscope.out': use '-f foo' in your Cscope invocation to name your database 'foo.out' instead, and you'll be OK.

12. That's it! Use ":help cscope" (in Vim) and/or "man cscope" (from your shell) if you've got questions, and to learn the fine points.

Using Cscope on large projects (example: the Linux kernel)
Cscope can be a particularly useful tool if you need to wade into a large code base. You can save yourself a lot of time by being able to do fast, targeted searches rather than randomly grepping through the source files by hand (especially since grep starts to take a while with a truly large code base).

In this tutorial you'll learn how to set up Cscope with a large project. We'll use as our example the Linux kernel source code, but the basic steps are the same for any other large project, including C++ or Java projects.

1. Get the source. First get the source code. You can download the Linux kernel source from http://www.kernel.org. For the rest of this tutorial, I'll assume you've downloaded Linux 2.4.18 and installed it into /home/jru/linux-2.4.18.

Note: Make sure you've got enough disk space: the kernel tarball alone is 30 MB, it expands into 150 MB of source code, and the Cscope database we'll generate will gobble up another 20-100+ MB (depending on how much of the kernel code you decide to include in the database). You can put the Cscope database on a different disk partition than the source code if you need to.

2. Figure out where you want to put your Cscope database files. I'll assume you'll use /home/jru/cscope as the directory to store your database and associated files.

3. Generate cscope.files with a list of files to be scanned. For some projects, you may want to include every C source file in the project's directories in your Cscope database. In that case you can skip this step, and just use 'cscope -R' in the project's top-level directory to build your Cscope database. But if there's some code that you wish to exclude, and/or your project contains C++ or Java source code (by default Cscope only parses files with the .c, .h, .y, or .l extensions), you'll need to generate a file called cscope.files, which should contain the name of all files that you wish to have Cscope scan (one file name per line).

You'll probably want to use absolute paths (at least if you're planning to use the Cscope database within an editor), so that you can use the database from directories other than the one you create. The commands I show will first cd to root, so that find prints out absolute paths.

For many projects, your find command may be as as simple as

cd /
find /my/project/dir -name '*.java' >/my/cscope/dir/cscope.files

For the Linux kernel, it's a little trickier, since we want to exclude all the code in the docs and scripts directories, plus all of the architecture and assembly code for all chips except for the beloved Intel x86 (which I'm guessing is the architecture you're interested in). Additionally, I'm excluding all kernel driver code in this example (they more than double the amount of code to be parsed, which bloats the Cscope database, and they contain many duplicate definitions, which often makes searching harder. If you are interested in the driver code, omit the relevant line below, or modify it to print out only the driver files you're interested in):

LNX=/home/jru/linux-2.4.18
cd /
find $LNX -path "$LNX/arch/*" ! -path "$LNX/arch/i386*" -prune -o -path "$LNX/include/asm-*" ! -path "$LNX/include/asm-i386*" -prune -o -path "$LNX/tmp*" -prune -o -path "$LNX/Documentation*" -prune -o -path "$LNX/scripts*" -prune -o -path "$LNX/drivers*" -prune -o -name "*.[chxsS]" -print >/home/jru/cscope/cscope.files

While find commands can be a little tricky to write, for large projects they are much easier than editing a list of files manually, and you can also cut and paste a solution from someone else.

4. Generate the Cscope database. Now it's time to generate the Cscope database:

cd /home/jru/cscope # the directory with 'cscope.files'
cscope -b -q -k

The -b flag tells Cscope to just build the database, and not launch the Cscope GUI. The -q causes an additional, 'inverted index' file to be created, which makes searches run much faster for large databases. Finally, -k sets Cscope's 'kernel' mode--it will not look in /usr/include for any header files that are #included in your source files (this is mainly useful when you are using Cscope with operating system and/or C library source code, as we are here).

On my 900 MHz Pentium III system (with a standard IDE disk), parsing this subset of the Linux source takes only 12 seconds, and results in 3 files (cscope.out, cscope.in.out, and cscope.po.out) that take up a total of 25 megabytes.

5. Using the database. If you like to use vim or emacs/xemacs, I recommend that you learn how to run Cscope within one of these editors, which will allow you to run searches easily within your editor. We have a tutorial for Vim, and emacs users will of course be clever enough to figure everything out from the helpful comments in the cscope/contrib/xcscope/ directory of the Cscope distribution.

Otherwise, you can use the standalone Cscope curses-based GUI, which lets you run searches, then launch your favorite editor (i.e., whatever $EDITOR is set to in your environment, or 'vi' by default) to open on the exact line of the search result.

If you use the standalone Cscope browser, make sure to invoke it via

cscope -d

This tells Cscope not to regenerate the database. Otherwise you'll have to wait while Cscope checks for modified files, which can take a while for large projects, even when no files have changed. If you accidentally run 'cscope', without any flags, you will also cause the database to be recreated from scratch without the fast index or kernel modes being used, so you'll probably need to rerun your original cscope command above to correctly recreate the database.

6. Regenerating the database when the source code changes.

If there are new files in your project, rerun your 'find' command to update cscope.files if you're using it.

Then simply invoke cscope the same way (and in the same directory) as you did to generate the database initially (i.e., cscope -b -q -k).

Friday, October 27, 2006

/proc/meminfo

Tips & Tricks
Featured Article: /proc/meminfo Explained

________________________________________________________________________
March 2003
________________________________________________________________________

"Free," "buffer," "swap," "dirty." What does it all mean? If you said,
"something to do with the Summer of '68", you may need a primer on
'meminfo'.

The entries in the /proc/meminfo can help explain what's going on with
your memory usage, if you know how to read it.

Example of "cat /proc/meminfo":

root: total: used: free: shared: buffers: cached:
Mem: 1055760384 1041887232 13873152 0 100417536 711233536
Swap: 1077501952 8540160 1068961792

MemTotal: 1031016 kB
MemFree: 13548 kB
MemShared: 0 kB
Buffers: 98064 kB
Cached: 692320 kB
SwapCached: 2244 kB
Active: 563112 kB
Inact_dirty: 309584 kB
Inact_clean: 79508 kB
Inact_target: 190440 kB
HighTotal: 130992 kB
HighFree: 1876 kB
LowTotal: 900024 kB
LowFree: 11672 kB
SwapTotal: 1052248 kB
SwapFree: 1043908 kB
Committed_AS: 332340 kB

The information comes in the form of both high-level and low-level
statistics. At the top you see a quick summary of the most common values
people would like to look at. Below you find the individual values we
will discuss. First we will discuss the high-level statistics.

High-Level Statistics
* MemTotal: Total usable ram (i.e. physical ram minus a few
reserved bits and the kernel binary code)
* MemFree: Is sum of LowFree+HighFree (overall stat)
* MemShared: 0; is here for compat reasons but always zero.
* Buffers: Memory in buffer cache. mostly useless as metric
nowadays
* Cached: Memory in the pagecache (diskcache) minus SwapCache
* SwapCache: Memory that once was swapped out, is swapped back in
but still also is in the swapfile (if memory is needed it
doesn't need to be swapped out AGAIN because it is already in
the swapfile. This saves I/O)
Detailed Level Statistics
VM Statistics
VM splits the cache pages into "active" and "inactive" memory. The idea
is that if you need memory and some cache needs to be sacrificed for
that, you take it from inactive since that's expected to be not used.
The vm checks what is used on a regular basis and moves stuff around.

When you use memory, the CPU sets a bit in the pagetable and the VM
checks that bit occasionally, and based on that, it can move pages back
to active. And within active there's an order of "longest ago not
used" (roughly, it's a little more complex in reality). The longest-ago
used ones can get moved to inactive. Inactive is split into two in the
above kernel (2.4.18-24.8.0). Some have it three.

* Active: Memory that has been used more recently and usually not
reclaimed unless absolutely necessary.
* Inact_dirty: Dirty means "might need writing to disk or swap."
Takes more work to free. Examples might be files that have not
been written to yet. They aren't written to memory too soon in
order to keep the I/O down. For instance, if you're writing
logs, it might be better to wait until you have a complete log
ready before sending it to disk.
* Inact_clean: Assumed to be easily freeable. The kernel will try
to keep some clean stuff around always to have a bit of
breathing room.
* Inact_target: Just a goal metric the kernel uses for making sure
there are enough inactive pages around. When exceeded, the
kernel will not do work to move pages from active to inactive. A
page can also get inactive in a few other ways, e.g. if you do a
long sequential I/O, the kernel assumes you're not going to use
that memory and makes it inactive preventively. So you can get
more inactive pages than the target because the kernel marks
some cache as "more likely to be never used" and lets it cheat
in the "last used" order.
Memory Statistics
* HighTotal: is the total amount of memory in the high region.
Highmem is all memory above (approx) 860MB of physical RAM.
Kernel uses indirect tricks to access the high memory region.
Data cache can go in this memory region.
* LowTotal: The total amount of non-highmem memory.
* LowFree: The amount of free memory of the low memory region.
This is the memory the kernel can address directly. All kernel
datastructures need to go into low memory.
* SwapTotal: Total amount of physical swap memory.
* SwapFree: Total amount of swap memory free.
* Committed_AS: An estimate of how much RAM you would need to make
a 99.99% guarantee that there never is OOM (out of memory) for
this workload. Normally the kernel will overcommit memory. That
means, say you do a 1GB malloc, nothing happens, really. Only
when you start USING that malloc memory you will get real memory
on demand, and just as much as you use. So you sort of take a
mortgage and hope the bank doesn't go bust. Other cases might
include when you mmap a file that's shared only when you write
to it and you get a private copy of that data. While it normally
is shared between processes. The Committed_AS is a guesstimate
of how much RAM/swap you would need worst-case.

Blog Archive