Transparent IAM Front and Center

The transparent IAM project has stepped into the priority 1 position because of a deadline in early March for a test of number portability in France.

March 15 looms large - .us corporate tax returns are due then. Then April 15 follows for .us person tax returns. These are unyielding distractions.

The transparent IAM feature now sends and receives stringified IAMs in the ISUPinRDNIS string that is copied to the the Asterisk RDNIS channel variable. Transparent IAM processing is optional per trunk group. The last big thing to do before releasing the feature to field test is to develop a new outbound transparent IAM parser. This new parser is aware of how to pick apart the outbound transparent IAM and use the harvested parts to assemble a new IAM. Here is the process being considered. The term sIAM is short for "stringified IAM" which is another term for the transparent IAM.

1. CIC value is determined by trunk group specified by dialplan and
the outbound hunt algo. in ss7boost
2. message type is IAM
3. fixed mandatory variables are taken from the sIAM
4. called number taken from dialplan
5. calling number taken from dialplan
6. redirection parms built from RED parms in ISUPinRDNIS from dialplan
if present, else take from sIAM if present
7. generic number parm built from GEN parms in ISUPinRDNIS from
dialplan if present, else take from sIAM if present
8. add on all other optional variable length parms in sIAM that have
not been used or overlayed already

This plan has been reviewed and approved by a reliable partner and user in France so it's a good bet that this will be very close to what's needed.

New SMG Release and Configuration Change Proposal

There will be a new SMG release today or tomorrow. The follow entry from the release notes summarize what to expect:

Patch 32 (2009-01-28 We)
* call processing, ISUP Transmission Medium Requirement pass to/from
Asterisk CALLTYPE channel variable via the sigboost callstart message
in the bearer.capability field found in sigboost.h
* Sangoma driver 3.3.15.6 required
* CMM, bug fixed, a span was being reported as clear and ready when the logs
indicated the ckt reset requests were being ignored by the telco; fix made
sure the ckt report showed the ckts in reset under this condition
* smgcfg.py, bug fix, numeration of formated wanpipe xmtp2 interface names
corrected
* transparent IAM, new feature work in progress, code added that does not
affect operations yet

As mentioned in a recent post here, the current method of configuring the association of span/chan to CIC and trunk group is too restricted. A new method is needed that allows configuring CIC and trunk group for each individual span/chan. But this approach might mean having to fill out a table for 16 E1s for a total of 496 entries. Since it is rare that individual control of all 496 span/chans is needed, then supporting configuration language that allows table minimization will be useful.

An example of explicit per channel configuration would be as follows:

Span	Chan	CIC	Trunk Group
0	0	1024	0
0	1	1025	0
0	2	1026	0
0	5	1027	0
0	6	1028	0
0	7	1029	0

Here is an example of minimizing language that accomplishes the same thing:

Span	Chan	CIC	Trunk Group
0	0-2.5-7	1024	0

Here's a slightly different example done explicitly:

Span	Chan	CIC	Trunk Group
0	0	1024	0
0	1	1025	0
0	2	1026	0
0	5	1029	0
0	6	1030	0
0	7	1031	0

And the same configuration using minimizing language:

Span	Chan	CIC	Trunk Group
0	0-2	1024	0
0	5-7	1029	0

There will also need to be a similar configuration technique to assign specific span/chans SS7 linkset, link, and SLC (noting that in ss7box, the link is always equal to the SLC value). Any span/chan in the CIC/trunk group table must not appear in the SS7 linkset/link table, and vice-versa.

Span	Chan	linkset	link and SLC
0	15	0	0
2	15	1	0

There will be no table minimization language used with the linkset/link table at this time.

CMM Bug Fixed Tonight

Found that a span was being reported as clear and ready when the logs indicated the reset requests were being ignored by the telco. Problem found and fixed in the Circuit Maintenance Manager. Confirming tests should be completed tomorrow.

Sangoma SMG Cluster Working

Multiple Asterisk/SMG using a single ss7box access point is delivered and working. Field trials are now underway.

Working on Transparent IAM for ported number handling in several countries. This feature pushes a copy of an inbound IAM into the dialplan. If the dialplan decides that the next leg of the call is an SS7 outbound call and IAM transparency is needed, then the copy of the inbound IAM is passed down to ss7boost where it will be used to construct an outbound IAM that is very similar to the inbound IAM. This is done to preserve information in the IAM when a call is forwarded, redirected, or ported.

Working on Transmission Media Requirement being mapped to and from the dialplan CALLTYPE per call variable.

ISUP T9 is implemented in ss7boost now. It was being handled with the Dial function timeout but this solution failed to pass several conformance tests where the release cause was expected to be 19. The Dial timeout was consistently delivering 16 as the release cause.

A major rework of the span/chan assignment of CIC and trunk group is planned. Currently simplifying assumptions are made about these assignments in a effort to reduce work done during configuration. The loss of flexibility from the current system is no longer tolerable, so a new system where each span/chan is explicited assigned a CIC and trunk group will be implemented.

A new configuration system is being rolled out where all SMG configuration parameters for wanpipes, MTP2, MTP3, and ISUP are entered into a single spreadsheet. The spreadsheet is then converted into a .csv file and fed into a new tool called smgcfg.py. This tool converts the information in the .csv file into the .conf files expected by the various component programs in SMG. The longer vision is to manage system configuration at a level above the spreadsheet so that engineers are specifying configurations using standard terms and language labels instead of using tables and indexes and coordinating information. Eventually, the tools will decompose humanly understandable configuration terms into the .conf file terms expected by the programs.

Product plans for 2009 for the SS7 part of Sangoma SMG include redundant ss7box, continuous operation upgrades, and SIGTRAN support.

A Second Quarter of Reliability, Performance, and Capacity

It's been a while since the last entry. Since that time the focus has been on managing 500+ circuits when four critical resources are lost and recovered. Those four resources are: signal gateway (ss7boxd), media gateway (sangoma_mgd), SS7 route, and bearer channel spans (E1 or T1).

Initial designs were found to be flawed when taken to high capacity test and some field trials. Two revisions followed and now the basic design is settled. The system, called Circuit Maintenance Manager (CMM) is now undergoing high capacity test and field trials. It currently supports loss/recovery of signal gateway, media gateway, and SS7 routes. The bearer channel span management function is ready for development now that underlying support is available from the Sangoma device driver as of this past week.

The CDR Logger is under going a change. The points in ss7boost where information is injected to the logger has been reduced and is now coupled tightly to the ss7boost message input/output functions and timer function. This change ensures no loss of info to the logger. The call serial number field is no longer provided because it did not make sense in some situations. Log entries are correlated into circuit event using the call setup id , span/chan and timestamps for outbound calls, and span/chan/timestamps for inbound calls. The reason for the change is that CDR Logger began as a tool for creating call records only but has also been valuable as a protocol analyser. The older logger missed some messages events which were not important to creating CDRs but were important for protocol analysis. The new logger improves the analyser function while retaining the call record creation function.

Another activity this summer has been load testing with 16 E1 loaded fully with 30 second calls. A number of bugs were found and fixed and the overall quality has improved noticeably.

There are now two release versions of SMG: stable and beta. This is more work but the time has come to provide options for SMG users. Stable releases will only receive small fixes and now new features. Beta releases will contain all fixes and new features. Stable releases that are closed will be reproducible. This marks a significant change from the past when development was much faster and installations were smaller and more change tolerant. As SMG has become more stable, users have indicated the SMG must change in a more controlled manner.

SMG ANSI installations are growing and bring new challenges. The circuit continuity test appears to be used more often in these locations. The procedure is slightly different from the ITU method. The final stage of COT reception functionality for four-wire circuits is being completed to comply with requirements from a major western USA service provider.

The third quarter 2008 will see initial development, testing, and field trials of the long awaited clustering function where asterisk boxes are clustered around an SMG box (already available) and SMG boxes are clustered around an ss7box box. Following cluster development will be the re-introduction of ss7box redundancy.

Other features in the works are expanded conformance testing and expanded ISUP parameter decoding. In particular are the calling party category and the generic number parameters. Asterisk does not have a convenient container for the many ISUP parameters so SMG is sneaking the information in and out of Asterisk using a char long string.

New MTP and ISUP Features

SMG/ss7box has been accepted into a major telecom network in North America.

The ss7boost Circuit Maintenance Manager (CMM) is functional in the field now. It is managing the blocking and resetting of circuits with respect to ss7boost restart, and the presence and absence of the media gateway interface (sangoma_mgd). In the coming weeks CMM will be managing circuits with respect to the presence/absence of SS7 routes, the signaling gateway (ss7box), and trunks (E1/T1 facilities).

ss7box now supports Management Inhibit reception sufficient to pass the limited conformance tests being administered in ITU-land. Management Inhibiting is used in conformance testing and rarely in operations. The procedure is complicated in some configurations. More time is being devoted to developing features for conformance tests lately.

ss7box will be supporting the reception of MTP Restart procedure in the upcoming week.

ss7boost now supports GRS reception fully. The circuits being reset are actually reset. ss7boost also now supports the CQM/CQR procedure and the ANSI CCR process.

Group Reset Reception Processing

The time has finally come where it is no longer possible to avoid developing the ill-specified Group Reset (GRS). Our siege of the GRS was broken while SMG was undergoing conformance testing in Belgium. We knew this day would come and it's good that we could hold out as long as we did, because the time has allowed the general approach to circuit maintenance in ss7boost to mature and improve to handle many circuits under stressful conditions. We are ready to implement GRS reception. We continue to resist implementing GRS sending for reasons we offer in detail at the end of this article.

The base ss7boost 1.0.2.x is in lab test and has passed calls in both directions. This revision has the first working version of the Circuit Maintenance Manager for queueing and executing requests for circuit reset, block, and unblock. The CMM is required as the base for developing GRS reception processing to ensure compatibility with the new circuit maintenance model in ss7boost.

GRS reception processor decomposes the request into individual circuit reset requests that are each then submitted to the CMM. The CMM queues the requests and manages sending the reset requests to the core circuit reset reception handler (CCRH). The CCRH performs maintenance actions and then passes the request to call processing where the the concerned circuit is handled for reset based on its current state. The final reset action in the CCRH is to acknowledge the completion of the circuit reset. Currently the ackowledgement is a Release Complete (RLC) message sent in response to a Reset Circuit (RSC). ss7boost will be modified to conditionally send an RLC or a completion indication to the GRS processor. This approach will eliminate the need to modify the call processing state machine, which eliminates a significant amount of regression testing to a core function that has approximately 1.1 million hours of field verification time.

Our dislike of the GRS comes from the awkward specification of the Range and Status parameter. The format and coding of the parameter changes if it is being used in a GRS, CGB, or CGU, It also changes if it is being used in ANSI or ITU SS7 flavors. In some instances, it allows up to 30-ish circuits to be affected in a group of up to 256 circuits. This means that it's possible to receive a GRS with a status containing a 256 bit status field with each bit needing to be test for 0 (not affected) or 1 (affected). The Range field assigns meaning to the values 0, 1, and n>1, and if n>1 is used, then the actual count of affected circuits in the Range is n-1. The GRS processor must search of allocate a variable number of octets in the Range. The Range and Status field has a very high degree of variability which drives up the numbers of exceptions and test cases. A great deal of code must be written and tested for the exceptions. The possibility of partial success exists with the GRS, the processing of which is not specified clearly.

The cost versus the benefit of the GRS in a small and distrubuted ISUP implementation like ss7boost is not favorable. Using the RSC for each individual circuit make smuch more sense because the implementation is straightforward. In a large switch with large trunk groups, the GRS makes sense. Most samples of GRS harvested from the field show a restricted and rational use of the GRS in that circuit groups are in the same E1/T1. It is our opinion that the idea of the GRS is good, the specification is too broad and uses tricks to save a bit of space.

Our implementation of the GRS reception will be based on our field samples and a sub-set of the specifications that minimizes the number of exceptions that will be handled. If our GRS reception processor encounters a GRS that is legally forms and yet exceeds the limits we impose, it will rejected and announced. We will consider remedies if this should occur, but we are betting it will never occur.

Merging ss7boost Development Streams

For a while now there have been two ss7boost code streams: stable and development. The stable stream is a repository branch and the development stream is the repository tip. The development stream insulated users from the difficulties of building the Circuit Maintenance Manager. The development stream has reached functional parity with the stable stream in terms of circuit management, and the development stream needs to be tested in the field. The time has come to merge the stable and dev ss7boost streams.

The merge is complicated because the two streams have been apart for several months. A wholesale "diff -up stable dev > patchfile" followed by a "patch -p0 <>

ss7box and MTP3 Route Management

ss7box has a partial implementation of MTP3 route management. Route management is needed when ever SS7 linksets are connected to a signal transfer point (STP). The ss7box implementation needs to be advanced for passing network acceptance testing, automatic restart, and improved fault tolerance. Up to this point, ss7box has not been connected to an STP where route management has been a critical issue. STPs tolerate nodes that ignore route management messages which also supports why the implementation is incomplete. As is often the case with ss7box, things changed suddenly this week, and now having a more complete route management is important.

Route management in MTP3 consists of procedures and messages per ITU Q.704 Section 13 Signalling Route Management:

• TFA - allow traffic to route
• TFP - prohibit
• TFR - restrict, don't allow unless no better choice to reach destination
• TFC - controlled, for communicating congestion levels
• SRST - signal route set test for asking and STP about the status of a route
• SRSC - signal route set congestion test for asking an STP about the congestion status of a route

In ss7box, route management is already designed and implemented at the core of the ss7box routing mechanism. The route management values assigned to each linkset are initialized to TFA and are never changed, but the values are checked for every outbound MSU passing through ss7box. Inbound route management messages are being decoded. These functions have been thoroughly tested in the field.

Inbound route management messages need to affect the route table and the values from the route table need to affect the related status of each linkset. If a route becomes completely unreachable, ss7box needs to send out a Destination UNAvailable (DUNA) M3UA message to all connected ss7boost clients. Conversely if a destination becomes reachable, a Destination AVAvailable (DAVA) M3UA message needs to be sent.

A signaling route set test procedure needs to be implemented to periodically query an STP that has prohibited a destination on its linksets to see if the prohibited destination is available.

A tool is needed to inject and receive route management messages for unit testing in the lab. This will most likely take the form of ss7box_cli commands for injection. In the lab two ss7box will be connected back-to-back. One will be the device under test (DUT) and the other will be the tester. The cli commands will be issued on the tester to the DUT. Messages from the DUT will be captured and decoded n the tester using normal ss7box route management procedures with some added debugging code.

The route set congestion messages and procedures will not be supported at this time as they are options in the spec and not frequently encountered.

Bug Found in ss7boost_cli --ckt-block

A bug was found with the ss7boost CLI commands:

./ss7boost_cli --ckt-block --span X --chan all
./ss7boost_cli --ckt-unblock --span X --chan all

The logs for the unblock follow:

Jan 31 22:46:08 acc05 ss7boost[9831]: I:sb_mpc.c:sg_bsm_unblock_sending:MARK:0
Jan 31 22:46:08 acc05 ss7boost[9831]: I:sb_mpc.c:sg_bsm_wait_uba:MARK:0
Jan 31 22:46:08 acc05 ss7boost[9831]: W:sb_mpc.c:sg_bsm_wait_uba:SS7 event id not expected:event id follows:2

Using block and unblock with individual circuits works:

./ss7boost_cli --ckt-block --span X --chan Y
./ss7boost_cli --ckt-unblock --span X --chan Y

This problem will be fixed in next major revision of ss7boost because the entire circuit maintenance method has been redesigned as mentioned in the previous post on this website.

No fix will be made in the following revisions and below:

Xyganda Technology, Inc. ss7boost Revision 1.0.1-25 (and any 25 branch revs like 25.2+)
Xygnada Technology, Inc. ss7boost Command Line Interface, Version: 1.0.1-01

Circuit Maintenance Manager and Bug Fixes

The maintenance of voice circuits in ISUP in ss7boost is being overhauled. Currently, call processing and circuit maintenance initiate circuit resets and blocks directly without any coordination. As hardware blocking design evolved it became apparent that circuit maintenance needed centralized management. The central maintenance manager (CMM) controls the resetting and blocking of all ISUP voice circuits - 496 maximum presently - per node. Each circuit has a maintenance action request queue. The CMM is dormant until queues are loaded and a trigger is pulled. The CMM checks all queues processing N circuits every M seconds until all queues are empty. Then the CMM returns to its dormant state.

The CMM code is written to a point that initial unit testing has begun. The number of test cases needed to verify operation looks to be high so unit testing with the ss7boost unit tester is a huge time saver.

There are a number of upgrades and configuration changes on-going so support work is spiking this week, which slows down CMM testing. This is exactly what makes predicting when large features will be ready very difficult. It's the nature of the business.

Speaking of unit testers, sangoma_mgd now has its own unit tester apparatus. A number of bugs got stomped this past December as a result of using it.

We caught and killed bugs in sangoma_mgd and ss7boost last week. In sangoma_mgd there was a case where a call ended and was not propogated to ISUP. In ss7boost we found that some sigboost messages from sangoma_mgd to ss7boost were not being properly entered in to the ISUP CDR log. Also in ss7boost we found that the "SMGRev-" prefix was not attached to inbound RDNIS strings in Asterisk/Callweaver. A new SMG release is out today that delivers the fixes - ftp://ftp.sangoma.com/linux/smg/smginstall-2008-01-21.tgz

New Look for ss7box.com

ss7box.com has a slightly new look. It is designed for rapid maintenance and moving as a defense against web hosts that suddenly and unexpectedly terminate service. It is also designed for optimal use as a technical user manual since that is its primary purpose. The design is embedded in a cascading style sheet.

We are back on blogspot

The wiki experience was OK until the web host went out of business. All the material is written in tikiwiki format and stored in an SQL database. I know next to nothing about tikiwiki administration and SQL databases. I also found that not all web hosts provide tikiwiki.

So, I revert to web 1.0. I will use HTML and docbook. The nice thing about docbook is that the same source will produce HTML or PDF. Since many ss7box users are charged per minute for Internet access, the wiki has not been the best documentation option for them. They are better served by downloading a PDF document that they can refer to repeatedly at no extra charge. Docbook is also a good system for porting content from one web host to another as they seem to be a transient.

New ss7box blog location

The ss7box blog has moved to the wiki because some people cannot reach the blogger.com website.

SMG High Capacity Testing

The Sangoma Signal Media Gateway continues to take a beating in the lab with five E1 continuously running at full capacity with short duration phone calls. Quite a few problems have been shaken out and the limits of the particular machines being used are showing. System engineering rules have also been developed as a result of this testing program. The test grow to support eight E1 and a third SMG in the near future.

The ss7box wiki is newly revamped with improved product plan sections. Have a look to see what's in store for SMG.

ss7mon will be called upon to examine working SS7 links as part of recent changes to our method of operation for replacing existing systems. Experience teaches that reported configuration information and actual configuration can differ. By using ss7mon we will be able to verify configuration parameters before swinging service to a new system. ss7mon will also let us measure the health of the existing system before putting a new system into service. ss7mon uses the same hardware as ss7box and uses a simple passive tap cable and couplers to access the working SS7 circuit. A brief service interruption is required to install the circuit tap.

ss7box Wiki Finds New Home

The ss7box wiki has a new home at none other than the ss7box website. Things are still being reassembled from various repositories. We are looking forward to using the new wiki. It's time to refresh the product plan.

SMG New Release

A new revision of Sangoma SMG is being released. This new revision is the result of extensive testing in the SMG bulk call generation lab in Toronto. This lab has proven itself capable of creating a punishing load similar to a unrelentless regional crisis. The systems under test were also misconfigured intentionally to increase the chance of circuit dual seizure occuring. In order to survive in the test, some parts of the SMG had to be redesigned and optimized. This was not surprising after it was done, but figuring out where the redesigns were needed was a difficult process. The latest release of SMG is solid foundation on which to continue building this product.

New installations are keeping us busy. The installation process in much closer to being fully automated now with the release of the smginstall tool in the latest SMG release. Some of our regional representatives are using the tool to successfully install and run SMG at customer sites.

Sadly we must report a HDD failure on the host of the ss7box Wiki. Alternatives are being considered and this tool should return to service before long. Backups were made. We are just not sure how fresh they are.

Currently we are working on expanding ISUP support to the lesser used messages. We are starting to see regional differences and operator differences. We are working with multiple linkset and multiple trunk group configurations now. We have a distributed SS7 presence experiment on-going and we are working on media gateway clustering capabilities. There's a lot to do but we are making forward progress daily and the results are tangible.

Bulk Call Testing Underway in Lab

The Sangoma SMG is undergoing bulk call generation testing in the lab. This battery of tests allows measurement of capacity, system load, endurance, accuracy, and reliability. In the past week since starting to use this test system, the problem find/fix rate has increased significantly. Also noticable is the elevated confidence felt by system designers and testers.

The bulk call tester was conceived and built in-house. It already has the ability to test Release 2 SMG systems having multiple networked call processing nodes. The system is easy to use and can be controlled remotely so that a geographically diverse development team can all control the test system as though they were in the same room with it.

Nanad Corbic, Chief Software Manager of Sangoma Technologies, Inc. says, "With the bulk call tester we expect to test SMG configurations handling 3000 or more circuits at very high levels of utilization. Already we see demand for SMG systems of this size."

SMP-Safe Queues Rule

Spinlocks are out, SMP-safe queues are in. All sites have been upgraded and have been running for several days without interruption.

Spinlocks, like any semaphore, should be avoided if possible - or so the conventional wisdom goes. And we understand a bit more about why it is considered conventional wisdom now.

SMP-safe queues have mutually exclusive read and write methods so that both operations can occur simultaneously. In our MTP2 implementation there is a single reader and writer for each of four queues. Three of the four queues are exposed to the potential for simultaneous reading and writing.

Announcing the Sangoma Signal Media Gateway

The SS7 ISUP attachment to Linux based softswitches finally has a name: Sangoma Signal Media Gateway. We call it SMG for short. Having a product name makes conversations and documenting so much easier. Here's a link to a product description: http://sangoma.editme.com/wanpipe-linux-asterisk-ss7

Installations in Pakistan, Paraguay, Manilla, Rome, and Michigan are helping us hone the installation process and are getting efficient at finding quality issues (bugs). They are pushing us further and faster along than we would have on our own. Thanks and keep up the good work.

The kernel module MTP2 had a set of nasty bugs that were cured with restructuring the buffer management scheme and surrounding a few critical section in the code with spinlocks.

We have introduced a trouble ticket system to track bugs. We give users access to the system so they can check up on our progress and make sure we are staying focused.

We found that we'll need to support overlap signalling in the early release of the SMG. Several locations cannot mandate en bloc signalling. The specs are a little weak in specifying how overlap signalling should work. Thanks to some attentive and knowledgeable users we captured the information we needed to complete the requirements we need to implement overlap signalling.

We've been told repeatedly that serving multiple E1 installations is required immediately. We identified the key item preventing us from delivering this capability is automated testing. We didn't want to deliver a multiple E1 support until we tested hundreds of simultaneous calls on a lab system. So, we designed and, are currently building, a bulk calling lab. We are going to replicate smaller versions of this lab in several places. The main lab will be able to generate thousands of simultaneous calls, verify connections, and vary hold times.