GSSAPI Overview

Categories: Programming, Cryptography


GSSAPI is a fairly old and well-established programming API for establishing a communications channel between two processes, including optional authentication and encryption.

In this article, the term “client” is used for the application which initiates/establishes communication and “server” is used for the application which waits for a client to connect to it.


The GSSAPI specification is language-independent; implementations exist for various languages and map the GSSAPI “design” into language-specific form. Being relatively old, the API is pretty ugly by modern standards - but effective.

The GSSAPI specification defines functions for channel init/setup, functions for encrypting/decrypting data (aka wrapping/unwrapping), and functions for signing/verifying data. It also defines the concept of a “mechanism” which is a kind of plugin which implements a specific handshake-sequence, and generates datapackets in some specific format. A particular concrete gssapi library provides a language-specific API, an implementation of that API, and bundles one or more mechanism implementations.

The channel-initialisation-function takes a “mechanism identifier”; an error is returned if the gssapi library implementation does not support that mechanism. The GSSAPI initialisation functions act as a kind of “state machine”; each call to the library returns a value which indicates what action the client application must take next - eg send the next output buffer to the receiver. This state-machine effectively “walks the client application” through the setup/handshake sequence of network packet exchanges. On the server side, it works similarly: when incoming data is first received, the server initialises a new channel, specifying the expected mechanism, and the gssapi library returncodes walk the server process through the necessary steps. Together this ensures the client and server perform whichever “handshake” is appropriate for that mechanism.

This generic state-machine approach allows a wide variety of mechanism initialisation-sequences to be supported through a common interface.

After setup is complete, the caller can use the GSSAPI wrap/unwrap functions to convert blocks of bytes to/from their encrypted formats.

At no time does the gssapi implementation perform any communications/network operations directly - that is the responsibility of the calling application. GSSAPI simply generates appropriate data-buffers and indicates when they should be sent. This allows GSSAPI to be used with any kind of communication system; GSS datapackets can be passed as RPC parameters, sent over a message-broker, send as emails, or transmitted via pigeons if desired! In practice, the underlying comms channel should be “low latency” as some mechanisms require multiple round-trips as part of the “handshake”; pigeons might not be appropriate after all :-)

The most-commonly-used gssapi mechanisms are for Kerberos, NTLM (now obsolete) and SPNEGO (described later).

GSSWrap and GSSMic

The GSSWrap function takes a block of unencrypted bytes and returns a buffer containing a “gsswrap data packet” that has headers and an encrypted form of the original data.

The GGSSMic function takes a block of data and returns a buffer containing a data-packet that has headers, the original data, and a “signature” (encrypted checksum). This functionality is not often used as encryption offers both integrity and privacy. Note: MIC = Message Integrity Check.

The gss-api-kerberos Mechanism

There is an rfc for the gss-api kerberos mechanism which defines a mechanism for setting up an authenticated and encrypted communications channel based on a Kerberos service ticket. Given an initial service-ticket (which includes a symmetric encryption key), the mechanism walks client and server through a sequence of states and generates a sequence of encoded buffers to agree that each is the expected party. The GSSAPI can then be used to encrypt packets of data for sending, and decrypt packets of data which have been received.

The “handshake” sequence for gss-api-kerberos is actually extremely simple, due to the fact that a Kerberos service ticket contains all the necessary data.

Both client and server must use a library that implement the gss-api-kerberos mechanism/protocol (ie the same state-machine and packet-formats) though they do not need to actually be GSSAPI implementations, ie the code does not need to offer the GSSAPI as its API.

A server that supports communication over gss-api-kerberos must listen on a dedicated port for that protocol.

The Kerberos specification defines packet-formats KRB_PRIV and KRB_SAFE, with the intention that client applications use these as containers for encrypted or signed data respectively - eg an app encrypts a block of N bytes with a kerberos symmetric key, then places it in the “payload” section of a KRB_PRIV packet, and sends it to a receiver. This can form the basis of an encrypted channel without using gssapi. However (unlike other parts of Kerberos) this functionality has never been widely used; in the vast majority of applications, transfer of data encrypted with a Kerberos session-key is performed via gss-api-kerberos.


As mentioned earlier, a gssapi library may support multiple “mechanisms”; the initialisation phase for GSS must specify which mechanism (state-machine and packet-formats) will be used. It is of course a runtime error if the library implementation being used does not happen to support that specified “mechanism”. And channel setup will fail if the server-side is expecting a different mechanism.

SPNEGO defines a mechanism for dynamically negotiating a suitable mechanism/protocol. A GSSAPI client application calls the GSS init function with SPNEGO as the mechanism; the underlying implementation then generates a special packet which includes a list of all protocols that the client supports, and indicates via the state-machine that this should be sent to the receiver. The server selects its preferred mechanism from the provided list, and sends this in a response message, which the client feeds back into GSSAPI. The spnego mechanism then delegates all further processing to the selected mechanism.

The result is that the client and server end up selecting a mutually-agreeable protocol, rather than having to configure the client and server with exactly one fixed common protocol.

SPNEGO is sometimes called a “meta-mechanism”, as it simply delegates to some other mechanism (eg gss-api-kerberos).

The generic state-machine design of GSSAPI makes this “pre-handshake handshake” phase transparent to applications which correctly use GSSAPI - it is just a couple of additional states in the state-machine.

Kerberos vs TLS

TLS is a “point-to-point tunnelling” protocol. It can be set up to:

  • authenticate both client and server, or just server, or neither;
  • encrypt or not encrypt the data.

However in the vast majority of cases, just the server side is authenticated, and encryption is enabled.

A Kerberos service ticket authenticates both client and server, and is therefore a better authentication method than TLS, when available. GSSAPI implementations often include both kerberos and TLS mechanisms; where Kerberos is available the client application should choose that in preference due to the mutual authentication.


GSSAPI functions are buffer-oriented; they take blocks of bytes as input and generate data-packets to be sent, and vice-versa. They also require the application using the API to implement the client-side part of the “state machine”, ie a loop which follows the instructions being returned from GSSAPI functions.

SASL is an alternative API which is “stream-oriented”. Like GSSAPI, it is an abstract API which supports multiple “mechanisms”. The SASL Kerberos mechanism uses the same on-the-wire format as gss-api-kerberos, and is in fact usually a wrapper over a gss-api-kerberos implementation. SASL implementations hide the “state machine” loop from users, making it friendlier to use.

SASL is considered the “simpler version” of GSSAPI, and does not have as much flexibility - or in other words, easier to use but not as powerful.

The most common mechanisms (ie “authentication/encryption protocol plugins”) used with SASL are:

  • EXTERNAL: used when setting up the socket already included authentication, eg using SASL over a TLS socket which has already done mutual-authentication-via-certificates.
  • PLAIN: a userid and password are passed in plaintext over the socket (which hopefully is encrypted)
  • NTLM: the old ms-windows login protocol
  • GSSAPI: this name is not well chosen; it is not “gss-api in general”, but “gss-api with the kerberos back-end and packet-encryption enabled”.
  • and various kinds of challenge/response protocols to authenticate via password but without passing the password itself over the network

Cyrus-sasl is the most popular SASL implementation for C. The Java standard library also includes an implementation.

Channel Binding

The GSSAPI documentation mentions “channel binding” in several places, but the description is difficult to understand. The concept has been given its own RFC, as it is actually a general-purpose concept.

As described in the channel-binding RFC, “the main goal of channel binding is to be able to delegate cryptographic session protection to network layers below the application” and “the critical problem .. is ensuring that there is no man-in-the-middle”. A common case is using Kerberos with a TLS or IPSec communications channel. Kerberos can be used to perform both authentication and encryption, but it is rather wasteful using CPU cycles to encrypt data that is being sent over an already-encrypted channel - particularly in the case of IPSec which can have very high-performance encryption and decryption. However TLS and IPSec don’t necessarily provide authentication themselves, ie an application knows that the data it sends cannot be intercepted but cannot be sure who it is talking to. This opens the door to a “man in the middle” attack, where the entity it is talking to is actually an attacker who is receiving all data, and then relaying it to the intended target system. Performing authentication (correctly) over such a network connection, then relying on encryption in the underlying connection is fairly pointless as the attacker can just let the authentication-related packets flow past and then start reading or modifying the data that is transferred after authentication is complete.

So the problem is: we can’t trust the authentication of an underlying connection but want to take advantage of its encryption. The answer is simply to gather some information about the endpoints and keys used by the underlying encrypted network connection, and include this information in the authentication process. This small amount of data is digitally-signed using the high-level and trusted authentication protocol (eg Kerberos), so a man-in-the-middle cannot interfere with it. And if both ends see the “same settings” for the underlying channel, then they can be sure there is no man-in-the-middle (ie there is just one connection, and not two hops) and can switch off any further encryption at the higher level, trusting the underlying channel to take care of that.

The datastructure transferred during a channel-binding is as follows:

  int32 initiator_addrtype
  byte[] initiator_address
  int32 acceptor_addrtype
  byte[] acceptor_address
  byte[] data

It shows the real addresses at each end of the system, and the “data” block contains the “settings” for the underlying channel, eg TLS or IPSec keys.

Framing and Magic Numbers

The databuffers generated by GSSAPI mechanisms may be transferred over various network channels, including TCP sockets, message-queues, and possibly even email. Some of these channels naturally indicate the “length” of datapackets, so gssapi mechanisms do not generally have “length” fields themselves. When transferring over TCP, it is therefore necessary to provide some way for the receiver of datapackets to know where each packet ends. This is called “framing” and is not part of the GSSAPI specification.

GSSAPI datapackets do, however, start with a two-byte “magic number” which at least makes detection of framing errors possible.

Java, GSSAPI and SASL.

The Java standard libraries provide an implementation of GSSAPI. This “Java flavor” of GSS is called JGSS; see package org.ietf.jgss. The official Java GSS documentation includes an example client/server pair that set up a channel and exchange some basic packets. This example code implements “framing” by writing (len, databuffer) pairs.

The java standard libraries also provide an implementation of SASL; see package

Other Interesting Information

NFS is a file-transfer protocol which is built on top of an RPC (remote-procedure-call) framework. NFS can use GSS to secure remote procedure calls; GSS tokens (data-packets) are passed as an RPC body.

There are some GSS mechanisms which provide authentication but not encryption. These mechanisms can usefully be combined with TLS. When TLS endpoints use self-signed certificates, then TLS can provide encryption but cannot provide authentication. GSS datapackets can then be exchanged over this channel to authenticate the endpoints thus validating the TLS channel.