DTMF tones, when transferred over RTP, will have the same issues. Different vendors try to overcome these issues by reading and implementing DTMF tone processors differently.

How Cisco sends DTMF tones...

Upon DTMF detection, Cisco GW starts sending tone updates immediately. It sends the initial packet 3 times, the initial tone duration is set to 0. Then it keeps sending updates every 50 ms until the end of the tone. Once the end of the tone is detected, Cisco GW sends the final packet 3 times with the correct “final” duration.

How Intel PIMG sends DTMF tones...

PIMG sends initial packet once but it starts the duration count from the next update. It sends updates every 20 ms, and the initial duration is set to 20. The next update will have duration 40 (even though the real duration at that time is somewhere between 20 and 40 ms). Once the tone is ended, the PIMG gateway sends 3 final packets with the "final" tone duration that is less (or equal) the duration that was reported in the previous update. (Very tricky: if you want to interop, you'll have to expect tone updates with greater sequence numbers but smaller duration).

I wonder, is it actually possible to implement this more different and still stay within the spec compliance...?

How to configure...

In the VoIP dial-peers set:

      dtmf-relay rtp-nte

This will use payload type 101 for the DTMF tones. If you want to redefine it, use “rtp payload-type” command in the dial-peer that requires this change.

There is a doc on the Cisco web site that covers these commands in details: “Dual Tone Multifrequency Relay for SIP calls Using Named Telephone Events” (http://www.cisco.com/en/US/products/sw/iosswrel/ps1839/products_feature_guide09186a0080087edb.html#58571)

I remember an issue when we were advertizing PT 96 in the outboung calls and Cisco GW was ignoring it and sending 200 OK with no support for DTMF. When turned on the "deb voip ivr all" and "deb voip ccapi inout", one of these debug outputs had messages about some application being bound to the payload type 96, and due to the conflict, it dropped the DTMF support for that call. Well, not arguing about validity of this (96..127 is the dynamic range after all), just redefine the map of these tones on the gateway, or use 101.

How to debug...

When DTMF tones seemed to be lost (completely not received) by one or another side, it’s very (very!) unlikely that the packet loss is the reason for this. Start from looking at the media negotiation. All or some messages of the call setup (Invite/Trying/Ringing/Ok/Ack) will have SDP block that should have something similar to this:

      m=audio 46090 RTP/AVP 0 8 101
      a=rtpmap:101 telephone-event/8000
      a=fmtp:101 0-16

SDP doesn’t really “negotiate” media parameters, but if you see INVITE that has “a=” with telephone-event, and 200 OK doesn’t have it, you’re likely to end up with the session that won’t use RFC 2833 for DTMF tones.

Does it mean that the DTMF tones are not transferred? It depends. In case with Cisco GWs, they will not extract DTMF tones from the audio, so the tones will come as a wave form. On the other end, if it’s a pure IP endpoint, it  likely won’t have tone waveform generator, and the tones won’t be sent at all.

If you want to see whether Cisco GW sends and receives tones (including all tome updates) use “deb voip rtp”. Here is an example of debug output:

Nov 10 07:49:29.061:  <<<Rcv> Pt:101    Evt:5       Pkt:0D 00 00
Nov 10 07:49:29.061:  <<<Rcv> Pt:101    Evt:5       Pkt:0D 00 00
Nov 10 07:49:29.065:  <<<Rcv> Pt:101    Evt:5       Pkt:0D 00 00
Nov 10 07:49:29.113:  <<<Rcv> Pt:101    Evt:5       Pkt:0D 01 90
Nov 10 07:49:29.165:  <<<Rcv> Pt:101    Evt:5       Pkt:0D 03 20
Nov 10 07:49:29.173:  <<<Rcv> Pt:101    Evt:5       Pkt:8D 03 88
Nov 10 07:49:29.181:  <<<Rcv> Pt:101    Evt:5       Pkt:8D 03 88
Nov 10 07:49:29.185:  <<<Rcv> Pt:101    Evt:5       Pkt:8D 03 88

How to read it: The “5” key was pressed for 113 ms (0x388 / 8). The first three packets are the tone start (3 times resend). The last three packets have the “End” bit on (Pkt:8D...) and they were also three times resent. The timestamp on each line of the logs help to see how the tone updates were sent.

If this happens and you already tried to access to your gateway through every interface configured, then the only choice left is using the console. Connect the Console port on your gateway with the COM port on the PC, open Hyperterminal, use the default settings for speed, parity and so on, once finished, press enter and check the prompt that the gateway is replying with. If you see something like this:

rommon 1 >

then your gateway failed to load and is currently in the ROMMON. If you’re curious what cause your gateway to get into this mode, use “sysret” command.

The main goal at this stage is to find the right IOS image and boot it. Check the devices available:

rommon 2 > dev
Devices in device table:
        id  name
    flash:  flash
bootflash:  boot flash

Check if flash: has any images already stored:
rommon 3 > dir flash:
         File size           Checksum   File name
  14423092 bytes (0xdc1434)   0x64cb5d98    c5400-is-mz.123-2.T5.bin
     13769 bytes (0x35c9)     0x229ec3b1    tippi5400-confg
     58095 bytes (0xe2ef)     0x8f9c2d6d    crashinfo_20000101-002059

This gateway already has 5400-is-mz.123-2.T5.bin, so we just boot it:
rommon 4 > boot flash: c5400-is-mz.123-2.T5.bin

If the image is correct and finished the booting successfully, you’ll get a regular prompt:
tippi5400>

Normally, in case of an internal failure or a power outage, the router/gateway should reboot and start automatically loading the saved startup configuration. Some basic post-mortem diagnostics of why router went down can be done by looking at the “sysret” output in the ROMMON or checking the output from “sh ver” once loaded (if you’re running one of the latest versions and it keeps crashing in similar conditions, you probably want to let Cisco TAC know about it, including sending them the crash dumps). The next step is to fix the boot procedure so the router could come up automatically.

The most likely reason for this problem is that the router/gateway couldn’t find the IOS image to boot. In the router’s config you can specify image(s) to load. In my case they were pointing to the wrong files in the flash: and caused the boot procedure to fail. Remove the obsolete/incorrect references with “no boot system flash <name>” and add the correct one with “boot system flash <name>”:
boot system flash c5400-is-mz.123-2.T5.bin

Verify that your changes work. Save the configuration and reboot, this will help avoiding surprises when your gateway dies next time.

So, I enter “wr” then “reload” and... this gateway doesn’t boot again. :-( Moreover, it doesn’t look like it was actually trying to boot from anywhere at all. So, I manually specified where to boot from, loaded again and checked if anybody messed up with the register. Somebody definitely did:

tippi5400#sh ver
Cisco Internetwork Operating System Software
IOS (tm) 5400 Software (C5400-IS-M), Version 12.3(2)T5,  RELEASE SOFTWARE (fc2)
<skipped>
Configuration register is 0x0

Well, it doesn’t load because it was specifically set not to. Fixed that by setting register to default with "config-register 0x2102". Again, saved and reloaded, and everything worked fine.

A couple of hints for getting rid of annoying messages:
“%Error opening tftp://255.255.255.255/tippi540.cfg (Timed out)” messages…
Enter “no service config” in the global config and these messages are gone.

“Translating "<something>"...domain server” when typing wrong command…
Enter “no ip domain-lookup” in the global config.

Finally, the gateway is up, so check that “no logging console” is set and forget about console terminal until next time you cannot telnet to your gateway.

There is a certain pain associated with downloading the IOS image from ROMMON via TFTP. In the lab gateway we laready had IOS image in the flash, so it was just the task of booting from it. Sometimes (especially after experiments with commands that erase the flash) your gateway may fail to boot because the flash is empty. Then you'll have to use ROMMON fro downloading IOS image as well. Use “set” command to specify IP_ADDRESS, IP_SUBNET_MASK, DEFAULT_GATEWAY, TFTP_SERVER, and TFTP_FILE, then enter “tftpdnld” which will download the image and place it into the flash (unless -r is specified). "reset" will cause gateway to reboot from ROMMON and verify if it can load successfully.

Do I need anything special configured for the call transfer?

• “application session” tells the gateway to use and application that “knows” how to transfer a call. By default, the “DEFAULT” application is used.
• “incoming called-number .” allows this dial-peer to be used as inbound dial-peer for any DNIS.
• “direct-inward-dial” allows you to avoid “assistance” from the gateway w.r.t. collecting additional information from the called like phone extension. Read “Understanding Direct-Inward-Dial (DID) on IOS Voice Digital (T1/E1) Interfaces” if you want to understand DID more.
  

Now all calls coming from the PSTN through this dial-peer will be processed by the “session” application and will be able to perform call xfer later.

Note: It seems that Cisco has added new command that helps to avoid configuring this “application session” for each inbound dial-peer: “call application global session”. I didn't try it yet.

Call transfer still doesn’t work. Why?

7-, 10-digit numbers and a prefix “9”…

Some time ago I ran into this issue with the outbound calls (and transfer as well). What appeared to be a misunderstanding between Cisco GW and PBX SL-100 (which our GW is connected to) with respect to the TON (TypeOfNumber) field for the Called Party Number, produced quite amazing ISDN Q.931 message exchange:

*Jun  6 01:04:04.735: ISDN Se3/0:23 Q931: TX -> SETUP pd = 8  callref = 0x0421
        Bearer Capability i = 0x8090A2
                Standard = CCITT
                Transer Capability = Speech
                Transfer Mode = Circuit
                Transfer Rate = 64 kbit/s
        Channel ID i = 0xA98397
                Exclusive, Channel 23
        Calling Party Number i = 0x2180, '4255554592'
                Plan:ISDN, Type:National
        Called Party Number i = 0xC1, '95557310'
                Plan:ISDN, Type:Subscriber(local)
*Jun  6 01:04:04.831: ISDN Se3/0:23 Q931: RX <- STATUS pd = 8  callref = 0x8421
        Cause i = 0x808B - Excessive digits received, call is proceeding
        Call State i = 0x01
*Jun  6 01:04:04.835: ISDN Se3/0:23 Q931: TX -> STATUS pd = 8  callref = 0x0421
        Cause i = 0x80E4 - Invalid information element contents
        Call State i = 0x01
*Jun  6 01:04:04.839: ISDN Se3/0:23 Q931: RX <- CALL_PROC pd = 8  callref = 0x8421
        Channel ID i = 0xA98397
                Exclusive, Channel 23
*Jun  6 01:04:04.843: ISDN Se3/0:23 Q931: RX <- DISCONNECT pd = 8  callref = 0x8421
        Cause i = 0x809C - Invalid number format (incomplete number)
*Jun  6 01:04:04.843: ISDN Se3/0:23 Q931: TX -> RELEASE pd = 8  callref = 0x0421
*Jun  6 01:04:04.879: ISDN Se3/0:23 Q931: RX <- RELEASE_COMP pd = 8  callref = 0x8421

So, in the Q.931 SETUP we provide 7-digit number with “9” prefix. The PBX didn’t really like the 8-digit number, complaining “Excessive digits received” in the Q.931 STATUS.

(Interestingly enough, Cisco GW didn’t like this STATUS message either, complaining back with “Invalid information element contents”. This status exchange is harmless – both sides just keep complaining :-).

I guess, PBX didn’t only complain about extra digits, it also cut them off. The DISCONNECT cause code says “incomplete number”.

There are two solutions to this:

1. Move switch type configured both on the Cisco GW and PBX to DMS-100. (I actually didn’t try this, only heard that as a recommendation).
2. Configure the Cisco GW to change TON to “Unknown”. Quite simple translation rule would help:
   (In the global config)
        translation-rule 1
         Rule 1 ^9 9 ANY unknown
   (In the outbound dial-peer)
         translate-outgoing called 1
   

Well, it you already got your gateway, you already have to know what kind of telephony link you’ll be working with because the hardware modules are different for analog and digital lines. If you have T1 (or E1) lines coming in, you have two choices for getting voice calls in and out: ISDN PRI and CAS (or Robbed bit signaling).

ISDN is expensive (takes 1 out of 24 channels in T1 for control messages and barely uses it, unless you configure NFAS) and complicated (w.r.t. the implementation) but it’s very simple to bring ISDN trunks up on the Cisco gateway. It's also easier to figure out what's going on in case of a failure.

CAS is cheaper since you get all 24 channels, but even getting the DNIS/ANI information becomes real pain. Another thing to watch out with CAS: those robbed bits are not exactly robbed, they are still in the stream and still have valid signaling information. For some types of signaling the flat silence won’t be "flat" anymore. If your receiving device has some adjusting mechanism for noise reduction, it may detect these periodic signaled bits and adjust accordingly (which may have greater impact on transferred voice than expected).

My personal experience with CAS: it never works the first time I plug it in. I find it much easier to deal with ISDN in the labs: can clearly see when call arrives, how it is handled and the tear down.

What should I know from my telephony provider about T1 lines?

Two things: framing and line coding. By default, Cisco has SF (SuperFrame) and AMI (Alternate Mark Inversion) for the T1 controller. However, almost every time the T1 lines we are getting from CO have ESF (Extended SuperFrame) and B8ZS (Bipolar with eight-Zero Substitution).
BTW, found quite interesting Q&A on this frame/line-code stuff, if somebody wants to understand that deeper: “Why B8ZS Isn't The Way To Go in Convergence”.

What happens if I make a mistake setting up framing or line-coding?

Well, for framing it’s simple: nothing will work if you try to use one type of framing instead of another. For line-coding it’s a little worse, because you may see line working but with errors. Once you configured the T1 controller, it makes sense to check if there are any errors on the line (use “sh contr t1” command).

What should I know from my telephony provider about ISDN line?

First, the type of ISDN switch. Cisco gateway has a variety of supported implementations:

sdn5350u(config)#isdn switch-type ?
  primary-4ess    Lucent 4ESS switch type for the U.S.
  primary-5ess    Lucent 5ESS switch type for the U.S.
  primary-dms100  Northern Telecom DMS-100 switch type for the U.S.
  primary-dpnss   DPNSS switch type for Europe
  primary-net5    NET5 switch type for UK, Europe, Asia and Australia
  primary-ni      National ISDN Switch type for the U.S.
  primary-ntt     NTT switch type for Japan
  primary-qsig    QSIG switch type
  primary-ts014   TS014 switch type for Australia (obsolete)

Usually the “NI” (National ISDN) is used, which I believe is supported by all switch vendors now. But this is definitely what you have to know from your provider.

What “B” and “D” channels mean?

It’s an ISDN terminology for marking the logical channels depending on the type of information they carry. A usual T1 ISDN PRI line will carry 24 channels 64 kbps each. 23 of them are “B” (Bearer) channels that carry voice/speech/data for a call, and 1 “D” (Data) channel carries control information about these calls (as well as some status information about these channels).

It's a little more complicated with european E1 PRI which has 30 "B" channels, 1 "D" channel and 1 synchronization channel (that's quite often not mentioned, but it still has a single timeslot allocated).

There is some confusion in the Cisco config related to the names and numbers of the D channel. When configuring the T1 controlled, you will have to specify a PRI group (that tells the gateway which timeslots to use):

controller T1 3/0
 framing esf
 linecode b8zs
 pri-group timeslots 1-24

This division into B and D channels happens in the above layer. Doen here, in the controller, these channels are identical timeslots. You can specify less number of channels (timeslots) to use (tricky, because the other side should be configured correspondingly as well), but you always will have the timeslot 24 (corresponding to D channel) configured. Even if you forgot and specified “1-3”, the gateway will do it for you automatically: “1-3,24”.

The next layer where you’ll refer to the D channel is a serial interface. Watch out, channels here are counted starting from 0, so the corresponding interface description will be “interface Serial3/0:23”.

Dealing with voice calls, the next abstraction layer in the config is voice ports. It still applies to the same D channel, but now it’s being referred this way: “voice-port 3/0:D”.
(I’m sure there must be a reason for this mess, but I think this is a natural way of regulating number of people that pass CCIE exams :-)

How to check if telephony side is up?

Well, run the following command: “sh isdn stat”. What you want to see is similar to this:

ISDN Serial3/0:23 interface
        dsl 0, interface ISDN Switchtype = primary-5ess
    Layer 1 Status:
        ACTIVE
    Layer 2 Status:
        TEI = 0, Ces = 1, SAPI = 0, State = MULTIPLE_FRAME_ESTABLISHED
    Layer 3 Status:
        0 Active Layer 3 Call(s)
    Active dsl 0 CCBs = 0
    The Free Channel Mask:  0x807FFFFF
    Number of L2 Discards = 0, L2 Session ID = 24

If MULTIPLE_FRAME_ESTABLISHED appears in the L2 state, everything is fine and now you can make calls and see them coming (unless the configuration of your PBX or CO is wrong and those calls are simply not routed via this T1 trunk).

If you see something like this:

ISDN Serial3/2:23 interface
        dsl 2, interface ISDN Switchtype = primary-5ess
    Layer 1 Status:
        DEACTIVATED
    Layer 2 Status:
        TEI = 0, Ces = 1, SAPI = 0, State = TEI_ASSIGNED
    Layer 3 Status:
        0 Active Layer 3 Call(s)
    Active dsl 2 CCBs = 0
    The Free Channel Mask:  0x00000000
    Number of L2 Discards = 0, L2 Session ID = 0

then you are likely to have a problem with the physical connection. Though the same result will be if you administratively shutdown this controller (using "shutdown" command):

or if you specified incorrect framing.

Hint: Gateways like any other hardware may fail and the ports may not come up because of the HW failure. But before blaming the HW, try a little trick, plug the working cable (the one that was ACTIVE on the Layer 1) into the supposedly failed port and wait for about 30 sec. If you see it coming up, then the port is OK. Unfortunately, if you see ACTIVE and TEI_ASSIGNED, then it is a config issue and will require further investigation.

If you want to get into the details of how gateway brings up the ISDN trunk, check the debug commands available under “deb isdn ?”.

I changed the (controller/interface/voice-port) config but nothing happen. What should I do?

Some things don’t get changed immediately. Restarting trunk takes time (somewhere around 30 sec before you see MULTIPLE_FRAME_ESTABLISHED if started successfully). If you turn on debugging, you will see the changes. I usually recommend restarting the particular controller or interface (using “sh”/”no sh” command) and closely monitor the debug output for errors or unexpected messages.

Actually, I’m playing with one of the controllers while writing this and ran into the same problem. The trunk was up and running. After “sh” and ”no sh” on the controller it went down and came back after a while. I did the same in the serial interface, but it went down and never came back. So, after restarting the controller again, everything came up. In other words, shake it here and there and at some point everything will start working... :-)

How to know which called/calling numbers are coming from PSTN?

I personally find the “deb isdn q931” command the most useful in cases when configuring the dial plan. Just after the trunks are up, enable “deb isdn q931” and make a call to the number that should be routed to this trunk. If nothing appears, check that you didn’t forget “term mon” command in your telnet window. But if still nothing appears, you may want to talk to your PBX guys because they seem to mess up their routing and calls just don't get here.

Normally, when making a call you would see something like this:

*May 19 03:17:01.222: ISDN Se3/1:23 Q931: RX <- SETUP pd = 8  callref = 0x077C
        Bearer Capability i = 0x8090A2
                Standard = CCITT
                Transer Capability = Speech
                Transfer Mode = Circuit
                Transfer Rate = 64 kbit/s
        Channel ID i = 0xA98395
                Exclusive, Channel 21
        Calling Party Number i = 0x2183, '4255551310'
                Plan:ISDN, Type:National
        Called Party Number i = 0xC1, '5553903'
                Plan:ISDN, Type:Subscriber(local)

(followed by more messages like CALL_PROC, CONNECT and other Q.931 messages). There are two things you may take from this debug spew: the called party number (or DNIS) and the calling party number (or ANI, or caller ID). Both fields (called information elements in ISDN) are optional and it’s possible that your PBX may give you any of them or none.

Actually, before configuring dial plan I usually check the phone numbers by making test calls and verify the following (don't expect them to be answered yet, just pay attention on SETUP message):

1. All numbers are routed to the gateway (i.e. each dialing ends up with incoming SETUP message on the gateway).
2. SETUP message contains correct information about called and calling numbers (keep in mind that both called and calling party number are optional information elements in the Q.931 SETUP message and may be omitted).
3. Check how many digits the called/calling number has (7 or 10, or something proprietary if it’s in the lab environment) and design the dial plan accordingly.

Note: It may get very tricky in case when your IP endpoint requires some kind of DNIS (for example, when selecting an application to run) and your PBX doesn’t provide one. I remember doing it using translation rules, translating any incoming DNIS (including no DNIS) into some artificial phone number for the app right in the inbound dial-peer. While browsing the available commands in the gateway, I saw several new commands that seem to be helpful in this case. I'll try them when I have a chance.

Can I figure out why the call was dropped from the Cause code in the Q.931 DISCONNECT message?

Not really. Though it gives you some ideas, but won’t help you figure out exact reason. Look:

*May 19 03:17:06.330: ISDN Se3/1:23 Q931: TX -> DISCONNECT pd = 8  callref = 0x877C
        Cause i = 0x80A6 - Network out of order
*May 19 03:17:06.418: ISDN Se3/1:23 Q931: RX <- RELEASE pd = 8  callref = 0x077C
*May 19 03:17:06.418: ISDN Se3/1:23 Q931: TX -> RELEASE_COMP pd = 8  callref = 0x877C

This call was disconnected because the remote endpoint has IPSec on (while the calling gateway doesn’t) and the attempt to establish a TCP connection failed. The “network out of order” doesn’t really help here. Interestingly enough, even when you see “Normal call clearing”, it doesn’t really mean that the call was successful. Welcome to the ISDN world! :-)

BTW, playing with the Cause codes in general is not very good idea. Since you have no direct control over the device that is responsible for translating this code into a message or a tone, you don’t really know what the caller will hear. The worst scenario in this case is dead silence (systems that don’t respond usually cause frustration and result into bad user experience) and that can happen if the provided code is unknown to that switch. You can redefine the cause code with “isdn network-failure-cause” on the serial interface if you want to ensure that all unexpected network errors will map into some known code (like fast busy, for example). Another command “isdn disconnect-cause” also overrides the cause code in cases when gateway failed to connect the call.

There is a long table with codes and descriptions published here: “ISDN Switch Types, Codes, and Values”.

Voice or data?

You have to remember to configure the serial interface so it could deal with the voice calls properly. Now, interestingly enough the command that configures this (under serial interface) has two options:

sdn5350u(config-if)#isdn incoming-voice ?
  data   Incoming voice calls will be handled as data.
  modem  Incoming voice calls will be handled as modems.

For voice calls you have to set to the “modem”, otherwise when calling in you’ll hear busy tone and the output from “deb isdn q931” will look similar to this:

*May 27 03:26:52.430: ISDN Se3/0:23 Q931: RX <- SETUP pd = 8  callref = 0x18B8
        Bearer Capability i = 0x8090A2
                Standard = CCITT
                Transer Capability = Speech
                Transfer Mode = Circuit
                Transfer Rate = 64 kbit/s
        Channel ID i = 0xA98391
                Exclusive, Channel 17
        Calling Party Number i = 0x2183, '4255558080'
                Plan:ISDN, Type:National
        Called Party Number i = 0xC1, '5552084'
                Plan:ISDN, Type:Subscriber(local)
*May 27 03:26:52.434: ISDN Se3/0:23 **ERROR**: call_incoming: call id 0x5AD0 call treated as DOV
*May 27 03:26:52.438: ISDN Se3/0:23 Q931: TX -> CALL_PROC pd = 8  callref = 0x98B8
        Channel ID i = 0xA98391
                Exclusive, Channel 17
*May 27 03:26:52.442: ISDN Se3/0:23 Q931: TX -> DISCONNECT pd = 8  callref = 0x98B8
        Cause i = 0x80AC - Requested circuit/channel not available
*May 27 03:26:52.526: ISDN Se3/0:23 Q931: RX <- RELEASE pd = 8  callref = 0x18B8
*May 27 03:26:52.526: ISDN Se3/0:23 Q931: TX -> RELEASE_COMP pd = 8  callref = 0x98B8