ISUP Circuits Stay in Quarantine State

> Dear Mr. ss7box,
>
>     Can you give me some insight into what theses messages are telling me:
>
> Nov  9 15:12:45 tele2ss7 sangoma_isupd[21182]:
> W:sb_cmm.c:cmm_quarantine_ckt:CQ_WAIT_TIMER:go back to
> CQ_WAIT_SG:span/chan/event_id/caller follow:10:22:1019:8
> Nov  9 15:12:45 tele2ss7 sangoma_isupd[21182]:
> W:sb_cmm.c:cmm_quarantine_ckt:CQ_WAIT_TIMER:go back to
> CQ_WAIT_SG:span/chan/event_id/caller follow:10:19:1019:8
> Nov  9 15:12:45 tele2ss7 sangoma_isupd[21182]:
>
> Puzzled

Dear Puzzled,

My first guess is that the remote side is not responding to our reset circuit messages. If this goes on long enough the circuit will be quarantined and stay in the quarantined state until the remote side answers our resets.

Sometimes this happens because the NI and PRIO values in MTP3 for ISUP on our side are not matched to those of the remote side. You can check this by using /ss7box/ss7box_cli --show msu on and looking in /ss7box/msu.log. Examine the inbound and outbound messages for a particular point code pair and ensure that the NI and PRIO values are the same.

If the values are the same then you'll most likely need to work with the remote telco to find out why they are not responding to the circuit reset. The telco on the remote side may or may not care about helping you, and it may be a function of their ability to help you.

If the values are different then make necessary changes to the NI and PRIO  columsn for this trunk group in the ss7boost section (soon to be renamed the isupd section) of the configuration spreadsheet and use the smgcfg tool to make a new sangoma_isup.conf. Apply the new conf and see if problem is resolved.

Six SMG Project Status

Since June I've been building a set of six servers to run SMG. A major goal of this project is to deliver 6 SMG servers fully configured and tested so they are ready to run out of the box. Two of the boxes have shipped and presumably are doing well. The four remaining boxes have been a challenge. Two Intel motherboards had to be replaced. One of the two was replaced twice, and the second replacement was to an ASUS board which is now in my warranty replacement inventory.

I specified Intel CPU, fans, and motherboards thinking that a single brand system must have been system tested. My doubts started upon assembling the first system and seeing the Intel fan attachment mechanism causing the motherboard to bow slightly. Then I started getting random kernel oops, panics, and freezes on the 5th box. I had to go through the Intel RMA process to get a replacement but not without a fight. At first, Intel rejected my claim and said that I damaged the board, and offered to sell a new board to me at a reduced cost. After some discussion Intel kindly reversed their position and replaced the board at no cost to me.

As I was waiting for the Intel RMA, I purchased a 7th board to move the project ahead. This board worked for a few days and then began failing with the same random kernel oops, panic, and freeze. Since this purchase was recent I was able to have the board exchanged at Tiger Direct without any problem. I built up box 5 using board 7. It wasn't long before it failed similarly. When I returned to Tiger Direct for yet another replacement I found that they had sold out of this board completely. I exchanged it for an ASUS board and had to buy DDR2 memory because the Intel board used DDR3 memory.

The Intel RMA board arrived and has been working well. The ASUS board is on the shelf because I didn't want to spend time working with an odd board when the project is very late already. I purchased a Thermaltake fan when I bought the ASUS board. This fan causes no board deformation. I will replace the Intel fans in boxes 3-5 with Thermaltake fans on the hunch that fans causing board deformation is the root cause of the board problems, and to simply eliminate the undesirable deformation.

Box 6 has a failed power LED. This is an annoyance. Had I quality checked all components as soon as I received them I wold have been able to get a complete box exchange at the store. My options now are to RMA the box with the manufacterer, fix the LED, or purchase a new box.  I will attempt to fix the LED first.

Another unexpected problem became evident after building the first box.  The analog cards were long and top heavy and had a PCI-e1  connector.  I estimated that these boards might fail during shipping because of stresses on the connector.  I also observed the boards sagging under their own weight.  To alleviate the stress and sagging during shipping and operation, I devised and constructed a support sytem made of Lexan and aluminum shown here.

Early tests showed noticeable heat build-up in the boxes.  Investigation led us to understand that the three analog cards in each box dissapate a lot of heat continuously. All six boxes have case fans added to their specifications.

The original project plan called for placing two patch panels in a desktop rack. When the user received the first two boxes and one of the patch panel racks, he saw that having one patch panel per rack offered more flexibility.  The project will now include three additional racks for a total of six desktop racks: three 4U and three 2U.

The project plan has been enahnced to include a 4 box daisy-chain configuration and a 'Y' configuration.  The plan was also improved to include bulk call testing using SIPP.

The project documentation will use textual, graphical, and video media.

The current status of the project is:

1. boxes 1 and 2 are shipped and working on site
2. boxes 3-6
1. are built and showing signs of long term stability
2. analog lines are working
3. require daisy-chain and Y configuration building and testing
4. require bulk call testing
5. require shipping to site
3. box 6 requires power LED repair
4. box 3-5 require CPU fan change out
5. 2U patch panel acquisition and preparation
6. video capture and edit of setup, operation, installation, configuration
7. .pdf documentation
   

sangoma_sccpd Passes Heartbeat Test

The sangoma_sccpd daemon is running and heartbeating with ss7boxd:

[root@ana3 ss7box]# ./xps
10978 -15 ss7boxd
11531 -15 sangoma_isupd
11536 -5 sangoma_mgd
11613 -15 sangoma_sccpd
9891 0 asterisk

It is sending a heartbeat to ss7box. The heartbeats from isupd and sccpd are recognized by ss7boxd who sends a heartbeat ack to each entity. Hearbeats from isupd are ack'd to isupd, and heartbeats from sccpd are ack'd to sccpd. Here are some logs:

First we see the SS7 links are up:

Oct 29 16:38:41 ana3 ss7boxd[10978]: R:link util:ls 0:link 1:msu oc 34:tot oc 162000:util 0
Oct 29 16:38:41 ana3 ss7boxd[10978]: R:link util:ls 0:link 2:msu oc 34:tot oc 162000:util 0
Oct 29 16:38:41 ana3 ss7boxd[10978]: R:link util:ls 1:link 1:msu oc 34:tot oc 162000:util 0
Oct 29 16:38:41 ana3 ss7boxd[10978]: R:link util:ls 1:link 2:msu oc 34:tot oc 162000:util 0

Then we see an inbound heartbeat from isupd (SI=5):

Oct 29 16:38:43 ana3 ss7boxd[10978]: I:mtp3_l4_m3ua.c:handle_asp_state_msg:M3UA_MT_ASPSM_BEAT:si/index/msg-type:5:0:3

Next we see an inbound heartbeat from sccpd (SI=3):

Oct 29 16:38:46 ana3 ss7boxd[10978]: I:mtp3_l4_m3ua.c:handle_asp_state_msg:M3UA_MT_ASPSM_BEAT:si/index/msg-type:3:0:3

We then confirm that the heartbeat ack was sent from ss7boxd to sccpd:

Oct 29 16:38:46 ana3 sangoma_sccpd[11613]: I:core.c:analyse_sg_0_heartbeat_ack:MARK:0

And finally, we see the process repeating continuously:

[root@ana3 ss7box]# Oct 29 16:38:49 ana3 ss7boxd[10978]: I:mtp3_l4_m3ua.c:handle_asp_state_msg:M3UA_MT_ASPSM_BEAT:si/index/msg-type:5:0:3
Oct 29 16:38:51 ana3 ss7boxd[10978]: I:mtp3_l4_m3ua.c:handle_asp_state_msg:M3UA_MT_ASPSM_BEAT:si/index/msg-type:3:0:3
Oct 29 16:38:51 ana3 sangoma_sccpd[11613]: I:core.c:analyse_sg_0_heartbeat_ack:MARK:0
Oct 29 16:38:54 ana3 ss7boxd[10978]: I:mtp3_l4_m3ua.c:handle_asp_state_msg:M3UA_MT_ASPSM_BEAT:si/index/msg-type:5:0:3
Oct 29 16:38:56 ana3 ss7boxd[10978]: I:mtp3_l4_m3ua.c:handle_asp_state_msg:M3UA_MT_ASPSM_BEAT:si/index/msg-type:3:0:3
Oct 29 16:38:56 ana3 sangoma_sccpd[11613]: I:core.c:analyse_sg_0_heartbeat_ack:MARK:0
Oct 29 16:38:59 ana3 ss7boxd[10978]: I:mtp3_l4_m3ua.c:handle_asp_state_msg:M3UA_MT_ASPSM_BEAT:si/index/msg-type:5:0:3

Twick or Tweet

Twitter has allowed me to return. I suppose I'll continue using it for microblogging the daily stuff.

sccpd and it's sister isupd are the twin daemons that came from splitting ss7boost into two separate functions. This project has been crawling along since May of this year, but it's finally become a top priority. isupd is fully separated operational and is part of the most recent stable SMG release. sccpd will require a new revision of ss7boxd. Anybody wanting to try out the tip development releases of sccpd should expect some updates to ss7boxd along the way. We're planning on getting most of the ss7boxd changes completed soon so that disturbances at the MTP3 levels are minimized.

The four box 48 hours stability test is still running and will be complete in about 12 hours from now. So far, this is the longest run ever for all 4 boxes. Intel motherboards have been a challenge. Intel silicon is fine. My spare motherboard inventory has been converted to ASUS already.

Dead Bird

Twitter won't let me in. No more tweeted status updates for now. Guess I'll go back to old-fashioned blogging.

The four box, 48 hour stability test passed the 24 hour mark this morning.

I've got a Sony video camera for documenting SMG procedures. I downloaded a video file for the first time this morning and found that Windows Media Player would not play the sound. Then I downloaded VLC Media Player which just worked with the MPEG2-PS files produced by the camera. Now I'm looking at video editing tools starting with CyberLink PowerDirector. I don't want to spend much time with this, so the first tool that just works will be the winner. (Follow up. This was hard and not much fun, but I finally managed to produce a video. I think I know enough now to make a useful video.)

This afternoon I'll complete the lab setup for sccpd testing and hopefully complete the first sccpd tests.

Normalcy Returns

It's been crazy busy since June. We delivered ss7box redundancy, 32 E1/T1 SMG, and some significant bug fixes. We're also working on a couple of unusual special projects that have resisted efforts to keep them on plan. Mix in several personal catastrophes requiring on-going time demands, and presto: crazy busy.

Thank you to everyone that had to wait longer than expected for promises fulfilled and support.

The good news is that things are settling down. The SMG product is showing itself to be on solid ground and we are now committed and able to support releases without requiring users to jump up to using development tip releases. The special projects are headed toward completion and the long-promised sccpd project is back on track.

What lies ahead? More intuitive configuration methods for networks and nodes. A continuous high volume testing program with more variance and talk path testing. Increased focus on sccp and related applications. Exposure of protocol layers using API's.

Thinking vs. Debugging

It happened again. I was working on redundant ss7box yesterday. When I launched the second ss7box there were lots of route audit log messages. Odd because ISUP was not running so there should not have been any route audits. How could I have screwed up so badly?

I got up and walked away. I did not return for over twelve hours. In that time I realized that I was not clear on how to configure redundant ss7box yet. Maybe the configuration was screwed up.

This morning I confirmed that the problem appeared as soon as the new redundant ss7box was started. I looked closely at the ss7box configuration and found that the same port number for sockets to the ISUP layer had been used on both ss7boxes. I found a new configuration rule.

This find strengthened the thought I had yesterday that configuration definition is still too complicated and redundant. Getting the configuration correct is difficult for me, so it will be impossible for a normal user.

This experience demonstrates the importance of taking time to think as part of the debugging process. Setting breakpoints and adding print statements are important too. What's more, it's active debugging. It shows that you are doing something. On the other hand, thinking is passive. It's often done while you are doing something else. It looks like goofing off. It's not. It's probably the most powerful debugging technique that I have. I first heard about it in engineering school at the University of South Carolina from Dr. Pettus. I've found my own way to what he was talking about. It takes a while to get there.

Problems are usually not solved on a time line or according to a deadline. If they are, they can be forced and awkward. Taking time to think through a problem and letting the solution come naturally according to function of the brain doing the thinking is valuable. My brain does not arrive at solutions well under high stress. It won't find any solutions when there's a lack of stress. Finding a tolerable balance of stress is the key.

ss7box: New Features, New Lab

ss7box is getting a lot of attention lately. Three new features are being developed simultaneously: redundancy, sccp routing, and support for a fiber interface. To support this development, significant changes to the Xygnada lab are required. Figure 1 below shows the build plan.

Figure 1

The nodes ana3 and ana62 will host a mated pair of redundant ss7box acting as a single point code. ana3 will host Asterisk, SMG, ISUP, SCCP, and a CNAM application that is capable of being both a client and a server. ana19 will be a clustered ISUP node using SS7 services from ana3 and ana62 with a new twist - it will have a point code that differs from the redundant ss7box pair. It will also host Asterisk and SMG. ana17 will be a single node instance of asterisk/SMG/ss7box. The dt node will be dedicated to SCCP and related applications with CNAM client/server being the lead-off application. This node will also serve as a developer workstation and regression test platform to support integration of a fiber interface into ss7box. In the middle, nodes ana60 and ana61 provide MTP3 transfer services like those found in STPs.

The lab configuration creates quite a few functional interactions and raises the overall lab complexity so that we can get more test coverage and carry out development in several areas simultaneously. Changing the lab is labor intensive and work in other areas stops as a result, so it's not done as often as needed. In this case, we could no longer put off the pain of changing the lab because progress had come to a halt.

The ana3, ana62, ana60, ana61, and ana17 nodes had been working together prior to the change. The dt node has been under construction for a while. It took a while to find the right version of opensuse (10.2) to work with the fiber interface libraries. Then it took more time to figure out that its A102c interfaces are incompatible with the modern SMG/ss7box so an upgrade to A102SH was required. The SS7 linkset between dt and ana60 was put into service today. What remains is to put the dt-ana61 linkset into service, create ana19, and reconfigure ana62 into an SMG/ss7box node from its current status as an SMG-only clustered node.

Configuration Changes

We are moving away from hand editing SMG configuration files, and moving toward using the smgcfgXX.py tool. XX is the version. We are currently using 01 and working on 02 now.

The tool uses a .csv file as input and the output are the wanpipe, ss7box, and ss7boost conf files. The .csv file comes from the export function in any spreadsheet program.

We support googledocs spreadsheets:
http://spreadsheets.google.com/ccc?key=rzeNA2SoiXKhaKX7Y9t2keg&hl=en

The smgcfgXX.py program is a Python script and is included with each SMG release.

We are working on an improvement where the smgcfg.py script will read the revision number in the .csv file and use the corresponding smgcfgXX.py script to continue processing.

ss7box Redundancy Explained

Two ss7box are better than one as is usually the case in SS7 networking. If one fails or needs to be worked on, the other carries the load and there's no interruption in operations. The next question is, "How will that work?"

The first thing that's needed is a two-link linkset as shown in Figure 1 below. The links need to be in two separate E1/T1 spans. Each span is connected to Sangoma wanpipe port on a separate Linux box. Each Linux box is running an instance of ss7box. This is the F-Link setup most often found in ITU networks.

Figure 1

In Figure 2 the left hand SSP from Figure 1 is shown as an Sangoma SMG implementation using redundant ss7box. ss7box and SMG communicate using SCTP/IP. SMG communicates with Asterisk using TCP/IP and UDP/IP. SMG can receive from either ss7box at any time. SMG sends to the pair of ss7box using a load balancing scheme. If one of the links to an ss7box is lost, or if an ss7box is lost or is under maintenance, then the full signaling load is handled by the remaining ss7box. When the second ss7box returns to service, load is automatically balanced between both ss7box.

Figure 2

ANSI networks and some ITU networks use redundant signal transfer points (STPs) to interconnect signaling terminations like signal switching points (SSPs) as shown in Figure 3. In this type of network there will be a combined linkset between an SSP and a mated pair of STPs. A combined linkset, in its simplest form, is a pair of single-link linksets. The SSP connects to both linksets. Each STP in the mated pair connects to one of the two linksets. The same rules

Figure 3

In Figure 4 the SSP from Figure 3 is shown as an Sangoma SMG implementation. The same rules regarding load balance, fail-over, and fail-back listed for the F-link configuration above also apply to this A-link (or combined linkset) configuration.

Figure 4

Redundant ss7box; GRS and Configuration Improvements

ss7box redundancy is the next big feature in the works. Now that SMG clustering is in the field and working, some larger systems are being designed. As predicted, those system designers quickly spot a single ss7box as a single point of failure. The solution is to deploy two ss7box acting as a single entity - so called redundancy. If one ss7box is lost then operations continue without interruption.

A redundant ss7box configuration minimizes interruptions during upgrades because one ss7box can serve on-going operations while the other is off-line and being updated. Then the updated ss7box serves operations and other ss7box is updated. Finally, both updated ss7box are put online and normal redundant operations resume.

We found a flaw in GRS processing in the past two weeks. The fix required enhancing GRS processing so that each trunk group has its own GRS queue. The ehancement is used if GRS processing gets stuck in ss7boost and the remote end keeps repeating requests for GRS. In this scenario, GRS requests are queued and after N queued GRS, the GRS queue overflows. When this happens, the GRS queue is cleared, all circuits in the trunk group associated with the GRS queue are put into quarantine, and subsequent GRS for the trunk group are ignored until all circuits in the trunk group have exited quarantine. The persistent nature of the GRS protocol ensures that a GRS will arrive after the GRS queue clearing protocol has completed. As ss7boost matures, more of these complex, self-healing protocols are being implemented which make SMG more robust and reliable. I makes SMG easier to use and maintain for users and support staff.

The smgcfg.py script is helping users and support staff create, update, and maintain SMG configurations. With the introduction of SMG clustering it became evident that hand editing text configuration files was unwieldy. With smgcfg.py the entire configuration for a node is entered in a single spreadsheet tab. We are using Googledoc spreadsheets to enable editable sharing of SMG configuration spreadsheets on a worldwide basis.

Check out the Twitter feed on this blog. Small increments of progress are reported in that media stream

Transparent IAM Passes Unit Testing

The transparent IAM feature works in the unit test environment. We are now working on enlarging the ISUPinRDNIS buffer in the sigboost callstart message to 900 octets. This change also demands that we convert to passing dynamically sized callstarts. At the same time we are adding a new field to the sigboost messages to indicate the version of sigboost a particular message adheres to. This is a safety mechanism for when sigboost interfaces are used by parties outside the closely associated original developers (CAOD). More than likely it will serve the CAOD well too.

We're struggling with installing some 3rd party protocol stacks in opensuse 11.1. It appears that the stack maintainers have not kept their products in step with recent major Linux distros. The job of sorting out the mess falls to me and this is not my strong suit. I am happy to learn as I go but others waiting on me would prefer to forego the learning and move directly to the doing part. I agree with them but find myself unable to comply.

CNAM work is bubbling up to the top. This time around we will be able to finish the project. I am refreshing myself on the code and I'm devising what the first step is restart this dormant project.

Some new features are being added to the list. One is providing trunk group and circuit status up to the Asterisk command line. Another is to add a session id to inbound ISUP calls to improve management of callstop and callstopack messages that arrive late and out of context. Comparing the session ids of messages to session ids of circuits will improve identification of messages that are useful, useless and discardable, or nonsense. The improvement will be that some messages are identified as safely discardable instead of confusing. If a message is confusing it can lead to a circuit being quarantined, which means a valid call might be disconnected, and the circuit becomes unavailable for use for several minutes.

Transparent IAM Front and Center

The transparent IAM project has stepped into the priority 1 position because of a deadline in early March for a test of number portability in France.

March 15 looms large - .us corporate tax returns are due then. Then April 15 follows for .us person tax returns. These are unyielding distractions.

The transparent IAM feature now sends and receives stringified IAMs in the ISUPinRDNIS string that is copied to the the Asterisk RDNIS channel variable. Transparent IAM processing is optional per trunk group. The last big thing to do before releasing the feature to field test is to develop a new outbound transparent IAM parser. This new parser is aware of how to pick apart the outbound transparent IAM and use the harvested parts to assemble a new IAM. Here is the process being considered. The term sIAM is short for "stringified IAM" which is another term for the transparent IAM.

1. CIC value is determined by trunk group specified by dialplan and
the outbound hunt algo. in ss7boost
2. message type is IAM
3. fixed mandatory variables are taken from the sIAM
4. called number taken from dialplan
5. calling number taken from dialplan
6. redirection parms built from RED parms in ISUPinRDNIS from dialplan
if present, else take from sIAM if present
7. generic number parm built from GEN parms in ISUPinRDNIS from
dialplan if present, else take from sIAM if present
8. add on all other optional variable length parms in sIAM that have
not been used or overlayed already

This plan has been reviewed and approved by a reliable partner and user in France so it's a good bet that this will be very close to what's needed.

New SMG Release and Configuration Change Proposal

There will be a new SMG release today or tomorrow. The follow entry from the release notes summarize what to expect:

Patch 32 (2009-01-28 We)
* call processing, ISUP Transmission Medium Requirement pass to/from
Asterisk CALLTYPE channel variable via the sigboost callstart message
in the bearer.capability field found in sigboost.h
* Sangoma driver 3.3.15.6 required
* CMM, bug fixed, a span was being reported as clear and ready when the logs
indicated the ckt reset requests were being ignored by the telco; fix made
sure the ckt report showed the ckts in reset under this condition
* smgcfg.py, bug fix, numeration of formated wanpipe xmtp2 interface names
corrected
* transparent IAM, new feature work in progress, code added that does not
affect operations yet

As mentioned in a recent post here, the current method of configuring the association of span/chan to CIC and trunk group is too restricted. A new method is needed that allows configuring CIC and trunk group for each individual span/chan. But this approach might mean having to fill out a table for 16 E1s for a total of 496 entries. Since it is rare that individual control of all 496 span/chans is needed, then supporting configuration language that allows table minimization will be useful.

An example of explicit per channel configuration would be as follows:

Span	Chan	CIC	Trunk Group
0	0	1024	0
0	1	1025	0
0	2	1026	0
0	5	1027	0
0	6	1028	0
0	7	1029	0

Here is an example of minimizing language that accomplishes the same thing:

Span	Chan	CIC	Trunk Group
0	0-2.5-7	1024	0

Here's a slightly different example done explicitly:

Span	Chan	CIC	Trunk Group
0	0	1024	0
0	1	1025	0
0	2	1026	0
0	5	1029	0
0	6	1030	0
0	7	1031	0

And the same configuration using minimizing language:

Span	Chan	CIC	Trunk Group
0	0-2	1024	0
0	5-7	1029	0

There will also need to be a similar configuration technique to assign specific span/chans SS7 linkset, link, and SLC (noting that in ss7box, the link is always equal to the SLC value). Any span/chan in the CIC/trunk group table must not appear in the SS7 linkset/link table, and vice-versa.

Span	Chan	linkset	link and SLC
0	15	0	0
2	15	1	0

There will be no table minimization language used with the linkset/link table at this time.

CMM Bug Fixed Tonight

Found that a span was being reported as clear and ready when the logs indicated the reset requests were being ignored by the telco. Problem found and fixed in the Circuit Maintenance Manager. Confirming tests should be completed tomorrow.

Sangoma SMG Cluster Working

Multiple Asterisk/SMG using a single ss7box access point is delivered and working. Field trials are now underway.

Working on Transparent IAM for ported number handling in several countries. This feature pushes a copy of an inbound IAM into the dialplan. If the dialplan decides that the next leg of the call is an SS7 outbound call and IAM transparency is needed, then the copy of the inbound IAM is passed down to ss7boost where it will be used to construct an outbound IAM that is very similar to the inbound IAM. This is done to preserve information in the IAM when a call is forwarded, redirected, or ported.

Working on Transmission Media Requirement being mapped to and from the dialplan CALLTYPE per call variable.

ISUP T9 is implemented in ss7boost now. It was being handled with the Dial function timeout but this solution failed to pass several conformance tests where the release cause was expected to be 19. The Dial timeout was consistently delivering 16 as the release cause.

A major rework of the span/chan assignment of CIC and trunk group is planned. Currently simplifying assumptions are made about these assignments in a effort to reduce work done during configuration. The loss of flexibility from the current system is no longer tolerable, so a new system where each span/chan is explicited assigned a CIC and trunk group will be implemented.

A new configuration system is being rolled out where all SMG configuration parameters for wanpipes, MTP2, MTP3, and ISUP are entered into a single spreadsheet. The spreadsheet is then converted into a .csv file and fed into a new tool called smgcfg.py. This tool converts the information in the .csv file into the .conf files expected by the various component programs in SMG. The longer vision is to manage system configuration at a level above the spreadsheet so that engineers are specifying configurations using standard terms and language labels instead of using tables and indexes and coordinating information. Eventually, the tools will decompose humanly understandable configuration terms into the .conf file terms expected by the programs.

Product plans for 2009 for the SS7 part of Sangoma SMG include redundant ss7box, continuous operation upgrades, and SIGTRAN support.