isectd doesn't really have arguments per se, but it does have
a rather elaborate command language for configuration and administration.
Commands are not read from a configuration file but are instead sent
to isectd by using isdclient.
For instance, typing "isdclient -c help" would result in:
abort isectd calls abort() - creates a core file
[un]block clientid [un]blocks incoming transactions from clientid
buckets num secs set the number of buckets to 'num' and the bucket
size to 'secs' seconds
client id examine the client structure for client id id
clients displays the client list
clone workerid clones a worker id
config events n remember n events (listevents)
delete {worker|client} id
deletes a client or worker
halt isectd calls exit(1)
idle id forcibly idles the busy worker id
listevents [n] list interesting events
pause service | worker | all
pauses a worker (makes lazy) without killing it
services lists isectd's services
preempt [on|off] sets or reports preempting option
pri id n changes a client's priority to n
ready service | worker | all
unpauses a worker
reject force isectd to reject new client logins
reset resets statistics
send worker_id|service_name|all command:(reinit, status(TP workers only))
sends command to a worker, service or all services
service id newservice changes a client's service to newservice
shutdown fd | all shutdown the fd for sending
start service | all start processing messages for service
stop service | all stop processing messages for service
unblock clientid reverses "block clientid"
worker id examine the worker structure for worker id id
workers [lazy | idle | busy | all]
displays the worker queues
<nil> examine isdData
There are five types of commands: configuration, service control, worker control, client control, isectd control.
Services are fundamental to how isectd manages the relationships between clients and workers. The contract clients and workers have with each other begins with how they find each other. When clients connect to isectd they tell it what service they're interested in. Respectively, when workers are configured they are assigned to services.
isdclient -c "config service echo"
In the example above, "config service" is the command and "echo" is the service name being created. Services don't do much of anything on their own except exist. They provide an umbrella for workers to organize under.
Workers are the programs that do the work of the application. Isectd needs to know what service name to assign a worker to, what host computer it needs to run on, the path of the executable (program), and any additional parameters the program may require.
isdclient -c "config worker echo localhost /usr/local/bin/isdecho"
The format of the command (above) is:
config worker servicename hostname pathname [parameters]
servicename is the name of the service the worker will
be assigned to. It must already have been defined using
config service (above). For programmers, it's
important to understand the parameters passed to the worker
program will not start at argc == 1. The first few arguments
to workers are used to communicate important information for
them to rendezvous with isectd. Additional parameters for
application programmer use start at argv[ISDARGMAZ].
Of the more useful debugging features implemented in isectd during its creation is its ability to maintain an in-memory log of things its done. Though the types of events logged can not be controlled (at this time) the number of them can. Events are kept in a circular linked-list.
The command below set the number of events to be remembered at 100.
isdclient -c "config events 100"
isectd's response would be:
isdData.NumEvents=100
Events are viewed using the "listevents" command.
Purpose: lists the configured services
Format: services
Example:
isdclient -c "services"
service id
eat 3
echo 2
op 1
Purpose: Requests the workers assigned to a particular service (or all services) be started.
Format: start servicename|all
Example:
isdclient -c "start echo" Ok. isdclient -c "start all" Ok.
Whenever isectd starts or stops services it will log messages to stderr reporting when
the workers login. isdexecd also logs messages to its stderr indicating the
command lines and the subprocess IDs (pids).
Purpose: Stops the workers associated with a particular (or all) service(s).
Format: stop servicename|all
Example:
isdclient -c "stop echo" Ok. isdclient -c "stop all" Ok.
Purpose: Halts service without terminating workers
Format: pause servicename|all
Example:
isdclient -c "pause echo" paused worker->id = 1, worker->name = echo isdclient -c "pause all" paused worker->id = 2, worker->name = eat paused worker->id = 1, worker->name = echo
Sometimes it's useful to halt transaction processing for a single service or all services. After executing the command, all the workers should show-up in the LAZY list, though their processes won't have been terminated.
Purpose: Undo a previous "pause" command
Format: ready echo
Example:
isdclient -c "ready echo" Ok. isdclient -c "ready all" Ok.
This is easily where most programmers and system administrators will spend their time. Tuning the system has more to do with the numbers of workers than most anything else. Also, worker administration is probably the most likely place problems will occur since isectd does not have direct control over the execution of worker processes. Remember, they are often running on remote machines and are started by another program, isdexecd.
Purpose: Displays the worker queues
Format: workers [lazy|idle|busy]
An example doesn't fit to well in the documentation, and it's probably more important to understand what it's showing more so than recognizing what one looks like.
Without any commands, "workers" displays the configured workers on the lazy, idle, and busy queues. The LAZY queue lists workers that for some reasone are configured but are unable to accept messages. This may be because they're associated service is paused or perhaps they haven't connected to isectd.
Idle workers are have logged on to isectd (they've connected) but aren't currently busy doing anything.
Busy workers have connected and accepted a message. Workers remain busy until isectd receives a reply from them that indicates that no more messages (replies) are expected back from the worker for this unit of worker (isdHeader.more).
Specifying any of lazy, idle, or busy, administrators can list only those queues they're interested in.
Regardless which queue is being examined, the following fields are included: service, worker id, avail? hostname, host pid, xactions, avg. resp., last resp., restarts, client id, and fd.
The service is the service name the worker is processing transactions
for. As noted earlier, this is the same service name clients will request.
The worker id is a unique number assigned to each worker. The
ID is used in commands like clone and delete for singling out workers.
avail? indicates with a 'yes' or 'no' whether or not the worker is
available to accept transactions. Avail is typically no whenever the
worker hasn't logged in, has been pause, or has been blocked.
The hostname and host pid display the hostname that was
used to configure the worker. The pid is the process ID used on the
remote host to identify the worker.
NOTE: the hostname will likely change in the future from displaying the configured hostname to displaying the IP address or DNS name for the remote end of the socket.
The number of transactions, average response times, and last response time are displayed next. Fory BUSY workers, the last response time is a running clock on the current transaction.
restarts is a cound of how many times isectd has had to restart
the worker (or request it to restart). isectd will automatically attempt
to restart any worker that closes its end of the socket without
isectd's knowledge. This typically happens when the worker's pid is
terminated or the process abends. If the service or worker is paused
or blocked, isectd will not attempt to restart it.
client id is relevant only to busy workers. It reflects the
client for which the worker is busily processing.
The column fd is used for the shutdown command explained
in the "isectd control" sub section.
Purpose: examine detailed information about a specific worker
Format: worker worker_id
Example:
isdclient -c "worker 1"
worker->node.next = 0x08055eb0
worker->node.previous = 0xffffffff
worker->node.queue = 0x0805004c
worker->id = 1
worker->service_id = 2
worker->hostpid = 895
worker->name = "echo"
worker->host = "localhost"
worker->path = "/usr/local/bin/isdecho"
worker->args = ""
worker->Transactions = 1
worker->fromCount = 1
worker->toCount = 1
worker->restarts = 1
worker->stopped = 0
worker->up_not_log_in = 0
Most of the information is self-explanatory. But the node
information at the top shows the forward and backward
links to adjacent workers as well as a pointer to the head
of the queue. The values for fromCount and toCount
are recorded separately from Transactions becuase
some clients will send requests in pieces, and workers may
reply in pieces. Such is the case when a worker is created
that processes large files (like print jobs). The client may
send 2048-byte chunks one at a time. Each chunk, supposing
it was sent to this worker, increases the toCount by one.
NOTE: one of the things I'd love to add to this command is
the ability to examine either the last transaction sent or
the current one. In the case that workers behave poorly
(as in, they hang) it would be nifty to be able to get a
hex-dump of the last thing they were sent since it's sometimes
difficult to determine what input the worker was unable to handle.
Purpose: creates an identical worker from an existing one
Format: clone wid
isdclient -c 'clone 2' cloned 4 from 2
One of the benefits of having isectd in the middle is your system can be scaled while it's running. What is meant by that is the number of workers assigned to any service can be dynamically increased simply by cloning or configuring new ones.
When a worker is cloned, everything is the same as in the original worker; same host, pathname, arguments, service, everything! The only difference is the new worker is assigned a new unique worker ID and the transactions counters begin at zero.
Purpose: removes a worker from the configuration
Format: delete worker wid
isdclient -c "delete worker 3" worker 3 deleted
Before workers can be deleted they must be on the LAZY queue, which means they must first be stopped.
As easily as configurations can be dynamically scaled up using the "config worker" and "clone" commands, they can be scaled down. Oftentimes during the development of a worker (or the early beta testing of it) they are more easily debugged when only one is running. When only one is running attaching a debugger predictably catches the next transaction since there are no other workers for the scheduler to assign the transaction to.
A preferable way of doing this is to use the "service" command documented under the subsection "Client Control."
Purpose: move a worker from the IDLE to the LAZY queue
Format: pause wid|all
Example:
isdclient -c 'pause 2' paused worker->id = 2, worker->name = eat
Pausing a worker is a good way to stop isectd from sending transactions to a it without terminating the worker. To re-enable the worker use the 'ready' command.
Purpose: readies a previously 'paused' worker
Format: ready wid
Example:
isdclient -c 'ready 2' readied worker->id = 2, worker->name = eat
Purpose: forces a worker on the busy queue onto the idle queue
Format: idle 2
isdclient -c 'idle 2' idled worker 2
Forcing a worker onto the IDLE queue is not an operation to be taken lightly. There are times however, during the development of a new worker where the worker itself may have become confused over the state its in and has forgotten to reply back to isectd. A programmer may have determined that it is safe to "idle" the worker. In isectd determines the worker was busy on a client's transaction (which it probably was) it will send a message back to the client indicating the worker died. It does this by setting header.error to '-1' and the message to "PROTOCOL INTERVENTION".
The benefit of doing this vs. just killing the worker and letting isectd/isdexecd restart it are probably minimal.
Though this is one way of doing it, an easier way is actually to configure a separate service and assign a single worker to it. After that worker has started (and presumably, a debugger attached to it) the operator can assign a client to the debugged service using the "service" command. The service command (documented in the Client Control subsection) is a sneaky way of moving clients to other services.
Note: it is not adviseable to assign clients to workers that won't understand the messages sent to them. Doing so can confuse both or either of the client and the worker--which may make sense in situations where programmers or QA analysts are interested in checking how either process handles exceptions of that nature.
The isectd control commands are pretty straightforward. They are intended to aid in the runtime-control of the middleware daemon as well as to assist in tuning it.
Since the original creation of these commands various other pieces of data have been added to the structures that would probably be worth displaying but haven't been coded yet. Eventually those, too, will be gotten around to and more information will be available.
Purpose: displays the root isectd structure and its interesting members
Format: <nil> (don't send anything)
Example:
isdclient -c "" isdData.configfile = "" isdData.host = "isect" isdData.ServiceList = 3 isdData.PriList = 1 isdData.LazyWorkers = 0 isdData.IdleWorkers = 2 isdData.BusyWorkers = 0 isdData.socket = 4 isdData.port = 12417 isdData.req_socket = 5 isdData.req_port = 5501 isdData.NextClientID = 4 isdData.MaxUsers = 0 isdData.command_worker = "op" isdData.Events = 0 isdData.NumEvents = 0 isdData.NumBuckets = 6 isdData.BucketSecs = 10 isdData.PeakBucket = 0 isdData.Buckets = 0 0 0 0 0 0 €
configfile - obsolete.
host - The hostname of the computer running isectd - usually the alias "isectd".
ServiceList - The number of services defined
PriList - The number of clients on the PriList queue. This is the same number as the number of clients attached.
LazyWorkers, IdleWorkers, BusyWorkers - the number of workers on each of the Lazy, Idle, and Busy queues.
socket - the file descriptor assigned to the socket isectd is listening to for worker connection requests
port - the TCP port isectd was assigned randomly by the host that it listens to for worker connections. This is one of the arguments passed to each worker through isdexecd.
req_socket - the file descriptor assigned to the socket isectd is listening to for client connection requests.
req_port - the TCP port isectd was configured to use for client connection requests. This should be the same port number listed in /etc/services.
NextClientID - the next client ID to assign. Increments sequentially starting at 0.
MaxUsers - The number of simultaneous client connections isectd will allow. If the value is 0, isectd will calculate the number of permitted connection by calculating the number of available file descriptors less the number of worker connections less a mysterious number called TOMS_CONSTANT I invented myself.
command_worker - the name of the service isectd intercepts for its own commands.
Events - The number of events currently in the Events queue
NumEvents - The maximum number of events isectd will remember. It was always remember the last 'NumEvents' events.
NumBuckets - For histogramming isectd performance, this is the number of buckets isectd is remembering
BucketSecs - The size, in seconds, of each bucket. 12 five-second buckets is a minutes' worth of data.
PeakBucket - The highest transactions completed value since histograming was started.
Buckets - A list of NumBuckets integers showing the number of completed transactions during each BucketSecs window.
Purpose: Displays a list of isectd commands
Format: help
Example:
isdclient -c "help"
abort isectd calls abort() - creates a core file
[un]block clientid [un]blocks incoming transactions from clientid
buckets num secs set the number of buckets to 'num' and the bucket
size to 'secs' seconds
client id examine the client structure for client id id
clients displays the client list
clone workerid clones a worker id
config events n remember n events (listevents)
delete {worker|client} id
deletes a client or worker
halt isectd calls exit(1)
idle id forcibly idles the busy worker id
listevents [n] list interesting events
pause service | worker | all
pauses a worker (makes lazy) without killing it
services lists isectd's services
preempt [on|off] sets or reports preempting option
pri id n changes a client's priority to n
ready service | worker | all
unpauses a worker
reset resets statistics
send worker_id|service_name|all command:(reinit, status(TP workers only))
sends command to a worker, service or all services
service id newservice changes a client's service to newservice
shutdown fd | all shutdown the fd for sending
start service | all start processing messages for service
stop service | all stop processing messages for service
unblock clientid reverses "block clientid"
worker id examine the worker structure for worker id id
workers [lazy | idle | busy | all]
displays the worker queues
<nil> examine isdData
Actually, any command isectd doesn't recognize, like "razmafritz," will result in a display of understood commands.
Purpose: isectd calls abort()
Purpose: isectd calls exit(1)
Purpose: set the number and size of buckets for histogramming
Format: buckets num size
Example:
isdclient -c "buckets 12 5"
The above example specifies 12 buckets of five-seconds each.
Purpose: lists interesting events starting with the most recent
Format: listevents [num]
Example:
isdclient -c 'config events 20' isdData.NumEvents = 20 [do some stuff] isdclient -c 'listevents 10' len=14, seq=1, more=0 request from client 404 select() = 1 client: priority=2, hostpid=7243, id=404, loghost=localhost select() = 1 request from client 403 select() = 1 worker x0x8055478 write-ready select() = 1 len=7, seq=1, more=0
Before you can listevents, you must first configure isectd to start remembering them with the 'config events' command. The output of listevents is usually of little interest but to developers and programmers. In more interestingly configured isectds (than the one I'm testing with right now) listevents does a good job of showing why isectd did or did not schedule a transaction from a client, or what's happening with various workers.
Purpose: closes an socket for writing
Format: shutdown all|fd
Connected, two-sided TCP sockets allow the closing down of one direction without closing the other. Isectd can 'shutdown' its write-end without shutting down its read-end. Shutting down a socket (usually used on worker sockets) makes the worker think isectd is closing down, which makes the worker shutdown, which makes isectd detect an abnormal worker termination and attempt to restart it. If this doesn't happen when a socket is shutdown the worker is not listening for messages from isectd an may have become catatonic.
Go to the first, previous, next, last section, table of contents.