Mbingo Hospital: The Mayo of Cameroon!

In January of 2014 while attending a regional conference in Bamenda, Cameroon, I was asked to fix an imminent network problem at a large hospital an hour away. It had been down for over a week and they knew the problem was internal. I happened to guess what the problem was off the top of my head so just hours later with the help of Chris Jackson, it was working again. I was asked to return as soon as I could and teach their IT guys how to manage their fairly complex network.

I packed the family up a month later (then living in Banyo, Cameroon) and moved out there for a month commitment, taught a 2 week class to all the regional hospital IT guys, worked on the hospital's networks, servers and databases, and lots of little stuff.  Fourteen IT guys (and even a girl!!!) came from as far as 9 hrs away to attend my class.

They spent 2 weeks, 3 hrs a day (including Saturdays) listening to me babel on about subnet masks, parabolic antennas, and structured wiring.  I gave them a quiz everyday and even a final exam. They all tried very hard and did very well. I was impressed that they understood me as well as they did. They each received a certificate of completion. Two of the Mbingo Hospital IT guys and one expat, Jesse Paulsen, took the class and continued to work with me for the next month at Mbingo hospital.

 

The above diagram is the logical layout of the network while below is the physical layout.  I created these to help future IT guys (most likely voluntary expats) to understand the system as rapidly as possible.

We completely rebuilt the network. Mbingo Hospital is a massive hospital with over 300 beds, digital X-rays, Electronic Medical Records, case logs, and electronic billing. It has 24 routers/APs (even after it was greatly simplified), 12 switches, 12,000 ft of Cat5 cable, 5 servers and spans over ¾ of a square mile. There are over 275 devices on the network including each doctor's tablet. Needless to say, I worked continuously alongside the three other guys the entire time. We greatly simplified the network, removing stuff that had built up for the last decade without anyone redoing it. We removed 2,000 ft of cable that was causing loops and reorganized the rest so that future folks could more rapidly understand and make adjustments to it. The problem was that so many people had come and worked on it for short periods, but no one had spent the necessary time to completely reevaluate it. We trekked around the premises removing bad APs, realigned them, changed the frequencies, relocated them, optimized them, removed them, simplified them... Overall the wireless network was running considerable better. We added a Linux firewall to filter inappropriate content, block bandwidth hogs, and block facebook and other social media sites during work hours (extremely popular amongst some and not so much for others). We added an internal portal website which allowed the staff to more rapidly look up phone extensions, connect to internal servers, and test their connections.

We also created a program which would monitor all the devices on the network and alert when something went down. I presented my findings to the heads of the hospital and made some future recommendations which they seemed to take seriously. Overall I had a wonderful time, lost some weight, and felt some significant improvements to the hospital's network were made. Hopefully the training I did will allow these 14 Cameroonian IT guys to further improve the Cameroonian hospital system.

What I learned:

• Terrestrial Internet is unreliable even when you have the CEOs number on speed dial.  Mixing a satellite connection with terrestrial connections gives you the best of both words.  If politics or rain conditions take out your terrestrial connection, you've always got satellite at 10x the price and 1/10th the speed.  Having a satellite provider in the blend even at a slow speed can at least keep things moving.
• I did not feel I did a good job working with the local staff.  I tried. I really did, to involve them but deadlines and efficiency took over.  I would say I have a lot of patience but still greatly lacking for the task.  I have kept in contact with them and we have a good relationship.  They would like me to return once again to help install the CHIAS software.
• DOCUMENTATION.  I can't stress how important it is to properly document your work in Africa.  Since there will likely be dozens of future IT people coming along with tight schedules creating accurate, up-to-date documents of things can be a God-send for anyone volunteering.  The learning curve on a system of this size could take you the entire trip just to wrap your mind around how it is built much less make improvements.
• Medical information on the Internet is incredibly useful to Doctors.  I didn't realize how powerful some of these free sites are to doctors in  the US much less doctors in Africa.  Access to UpToDate, epocrates and emedicine to name a few can GREATLY increase the medical care provided.  Pass the Word!
• Never ever put a 2.4GHz wireless backbone in place.  There are way too few channels in that band.  I put a 5GHz system in at Banyo and I can only hope that someday we can upgrade Mbingo to 5 GHz.  We spent tons of hours using MetaGeek inSSIDer trying to fix overlap.  And that's with no noisy neighbors!
• Clearly label switches and routes that are using Spanning Tree Protocol so when future people replace a dead switch they will realize they are crashing the network by replacing it.  Label every wire, every time.
• Great programs used in this project: MetaGeek inSSIDer and Wifi Analyzer for Android for troubleshooting wireless problems.  Wireless management: Ubiquiti AirControl2, Ubiquiti Unifi Controller.  For documentation: Network Notepad, Google Earth, and spreadsheets

CHIAS Beta Release

The Challenge

After spending a month and a half in Mbingo teaching a networking class and listening to all the regional hospital's IT directors talk about their needs, I was off to create a solution.  What came out of it a month later is my first release of CHIAS (CBC Hospital Internet Access System), an add-in to IPFire that modifies it to do something it could never do before: make money...I mean make Internet Systems in Africa sustainable.  Surprisingly to me, there is no open-source software out there that really does this well in an African context.  Why you ask?  We don't use electronic money here.  Yep no credit cards, debit cards, PayPal, even salaries are all paid in cash.  While there are a ton of solutions out there that work great in a first world scenario because of this, few to none use good old fashion cash.  The flow would need to work something like this:  a potential user (patient, doctor, visitor, employee, resident) is made aware of the wireless/wired Internet system. They may attempt to connect and get on via wireless, but they will not be able to access the Internet because a captive portal page would come up. 

The Solution

In many places in Africa, hospitals, NGOs and the like have setup a low powered computer with 2 NICs that acts like a router/firewall.  Using free software such as IPCop, PFSense, IPFire, ClearOS or many others, they are able to have the benefits of a fancy firewall/router for the cost of an old computer.  While each of the above mentioned have their pluses and minuses I decided to use IPFire as my starting point.  The main advantage to IPFire is it has some powerful filtering and caching abilities that are straight forward to use.  The downside is that IPFire, from LFS (Linux From Scratch) is difficult to make changes to because it doesn't have a compiler built-in.  You'll need to compile on an external box.  BUT that's ok because we want as thin as possible anyway right?

I started with a partially complete open source product called PlayBilling.  It needed to be extensively modified but it had a lot of the basic components that I was looking for.  The way it actually handled the iptables was not at all going to work out of the box, but the database and administrative reports were perfect.  Lots of small modifications like language, currency, and format were modified as well to work in Cameroon. 

How it Works: The User Experience

When a user hops on to our wifi system, regardless of what site they try to go to, this page will come up:

An Intranet portal page comes up which gives the user access to locally served resources such as offline Khan Academy, phone directories, hospital information, etc all running locally on the IPFire box.  These all work regardless of an Internet connection.

A link to "login" is also visible. Upon clicking on the link information is presented that the system is available to them for a fee. In order to use the system they are instructed to go to the person at the hospital who takes payments and request access.

They would decide how much to prepay for a certain amount of credit which would get put on their newly created account. They would sign the acceptable usage policy form there and would then return to their computer with their username and password which would grant them access.

   Upon a successful login, the system then deducts the amount of time they use until they log out or until their account is out of money.
 

At this point they would need to put more money on their account in order to get back on.  An option is available for them to check their account activity to verify proper billing.

How it Works: Administration

Furthermore, the system has a backend interface which allows the administrator to modify the system.  They can setup a fee per minute of usage or differing fee based on time online.  We are currently charging 300CFA/hour or $.60.

The administrator can create new accounts,

 list members and edit each one (obfuscated for privacy),

Create reports summing the total usage for a month per user  (obfuscated for privacy)

 

Or create a report for a given user over a given period to check for any "I forgot to logout" fixes.

 

One of the main features that I added was the ability to allow certain websites through even if people don't authenticate.  In our environment, you want medical staff to be able to access life saving information on the Internet without worrying about cost.  We give the staff unlimited access to a bunch of sites that are extremely helpful if not live saving.  Double checking the doctors dosage (or challenging handwriting) for drugs on epocrates.com is critical to patient care.

Key Features:

• Transparent Proxy with filtering, caching, update acceleration and Authentication.  Nothing needs to be modified on the client device such as proxies or DHCP custom configurations.  The firewall either completely blocks the client device from entering the firewall or allows it.  The firewall then transparently does all the filtering, caching, virus scanning etc.
• Works with any device including iPhone, Android, PCs, Laptops and Tablets without modification.
• Prepay with cash before use Captive Portal for IPFire
• Alleviates pressure from people wanting to use your connection.  If you let one person on, then everyone will want on.  Most people who really need Internet know it is well worth $.60/hr. 
• Filter unwanted content, track users activities, block sites from different locations (using VLANs)
• Non-authenticated Unlimited Users-You can assign MAC addresses to bypass the authentication altogether but still keep filtering and tracking.
• Khan Academy running on IPFire box using KA-Lite.  If you don't know what Khan Academy is then it is well worth researching.  It is amazing and lots of people here use it, including me, to learn or polish up their education.  Running it offline and on a box that is already running 24x7 is a great way to make it available to all.
• Intranet Portal

Why?

So why is this such an important thing to have? In Africa bandwidth is expensive and a lot of connections are charged based on the total amount of data you transfer in a month. Many organizations don't allow people access to their system because it would probably put them out of business in bandwidth fees. A system like this is ideal because it allows access and also allows the system to scale with the demand. As more people use it, the slower it gets but the more money it raises. Administrators would then buy a larger connection with the money raised. A sustainable system is developed which is a win-win for everyone. Furthermore there is a significant efficiency gained through this system. IPFire has a download accelerator built in and a caching server which reduces and deduplicates downloads. For example, a computer downloading a Microsoft Update for example will have to pay the full price of it, but each subsequent user will be able to download the update straight off the firewall at LAN speeds. This not only reduces costs for the overall system but frees up limited WAN bandwidth. Frequently, updates will never finish.  The system will be stuck in an infinite loop trying to download updates that will never finish.

So Far:

I've been debugging the software for the last few months, trying to get a good feel if this is working well or not.  We had a rough start with some devices not being properly billed but a few modifications and they are all working well now.  I was surprised as to how in demand this system has become.  We were able to pay for 1/3 of our subscription cost from individuals paying for access.  This helps the hospital by reducing the cost of the system which they would normally have to front.  My hope is to distribute this to other interested hospitals but as I only have a few more months left in the field here, I am not interested in supporting it or updating it with future releases of IPFire.

Interested?

If you are interested in running this software please let me know.  I can send you my code but like I said, it is still in beta and this is all for IPFire only.  There is no installer so you'll need to know your way around Linux.  I do have an installation document but this isn't for the faint at heart!

 



Introduction: About Joshua Shinar

Myself and Marius in C.A.R. building a remote relay

 

Introduction:

My name is Joshua Shinar and I'm an electrical engineer living in Banyo, Adamawa, Cameroon and formally Gamboula, Central African Republic, Africa. I grew up in the United States, went to Georgia Institute of Technology (Georgia Tech) and graduated in 2002 with a BS in Electrical Engineering. I had the pleasure of working with Motorola for 3 years, then went off on my own starting 4 IT/cloud computing companies over the next 10 years. I was fortunate enough to have a buyer who wanted to buy all of them and consolidated them into one, now collectively known as Adopt Technologies. The companies were sold, my wife turned in her resignation to Mayo Clinic and we were free. We had been planning this for over 10 years now and we were ready.  We always wanted to work in the humanitarian sector and applied with the ECC to work in Central Africa. We had PLENTY of adventures but that's another blog (joshandlorisafricanstory.blogspot.com). This blog is focused on the engineering side of things here. I'm actually beginning this 2 years in, but better late than never. Hopefully those reading can use some of what I've learned to make Africa even a better place!