Archive for the ‘Technology Projects’ Category
Sunday, September 12th, 2010
As promised here’s the second issue we ran in to when trying to resolve the first.
To recap the last post, responses to the traffic coming in to the Tidziwe network through the secondary connection, were being sent back out through the primary connection, because of the default gateway. This wasn’t working, because the response traffic was being NAT’ed to a different address as it went back out through the primary connection. We were able to partially get around that by adding NAT entries on the primary router to NAT the source address on that traffic to the correct address for the secondary connection, even though it was really egressing through the primary connection.
This ugly fix appeared to work pretty well at first. From where I’m located, I was able to get to the publically accessible services through both the primary and secondary connections. Unfortunately, some of the people located in Malawi weren’t so lucky. In fact, anyone that used Tonse (the provider of the secondary connection at Tidziwe) as their ISP in Malawi was unable to access the webmail and other services through the secondary connection. This at first didn’t make sense – if anything it would seem logical that accessing it from the same ISP would have fewer problems, not more.
Long story short, Tonse was dropping the response traffic as is entered their network, which is a reasonable thing to do given the circumstances. The packets said they were coming from an address inside of Tonse’s network (2.2.2.2 in our example below). Yet, how could this be given that the packet was coming from Tonse’s upstream provider?
Unfortunately in the short term, we don’t have many options besides our current ugly hack. To get around this additional issue, I added a route map on the Cisco router (the default gateway). The route map routes outgoing traffic through the DLink if it is:
- Going to Tonse’s network, and.
- Coming from one of the services that can be accessed through the Tonse connection
Here’s what the configuration looks like:
int LAN-Interface
ip policy route-map Tonse_Services
access-list 185 remark Used for Tonse_Services route map
access-list 185 permit tcp host 10.1.1.1 eq 443 41.76.0.0 0.0.255.255
route-map Tonse_Services permit 10
match ip address 185
set ip next-hop IP-Of-DLink
So far, no major problems with this. Hopefully we’ll be able to use both connections through a single device in the future, helping us avoid these sort of issues altogether.
Tags: africa, cisco, malawi
Posted in Technology Projects, Trip | Comments Off
Monday, August 30th, 2010
We recently discovered what was causing some of the connection issues over the WiMAX connection in Malawi.
There were two issues, but one sort of builds on the other. The first issue occurs when someone accesses a publically accessible service through the IP from the secondary ISP, Tonse. We have two routers at this site. A Cisco 1800 series router is connected to the main satellite connection. We’re using a DLink VPN router on the Tonse connection, for a site to site VPN to another DLink VPN router at a satellite clinic.
The Cisco router is set as the default gateway on the LAN at the main clinic site. The Cisco router has no information
It only see’s the outgoing traffic, not any of the incoming traffic that goes through Tonse and the DLink router.
1. Users browses to AtMail webmail using Tonse address.
To: 2.2.2.2
From: 5.5.5.5
2. DLink translates destination IP address to internal server.
To: 10.1.1.1
From: 5.5.5.5
3. Internal server replies, sending traffic to default gateway (Cisco router).
To: 5.5.5.5
From: 10.1.1.1
4. Cisco router has no record of incoming connection, translates source address to Ariave IP.
To: 5.5.5.5
From: 1.1.1.1
5. End user’s computer receives the response, but it is not from the address it connected to. The TCP/IP stack does not see this is a response to the TCP connection request, so it discards it and the connect is never established.
This is rather ugly because the Cisco router sends traffic out with a source IP address that doesn’t belong to it (and, is in fact on a completely separate network and service provider from the one it is connected to). It is also rather ugly because it is unclear (at least to me), how the Cisco router is choosing which NAT entry to use for translating the outgoing traffic.A ugly work-around for this issue is to add NAT entries on the default gateway (the Cisco router in this case). So, in addition to have the service NAT’ed to the public IP the router itself uses, it also has a second entry in the NAT table connecting the same internal IP / port to the public IP the other router/connection uses. This appears to cause the Cisco router to to translate the source address of the outgoing traffic to the public address of the other router/connection if it is not part of an existing connection that already came in through the Cisco router.
Doing this however caused the second issue, which I’ll post about in part 2. It’s a great example of why you shouldn’t do things the “ugly way” if at all possible. Stay tuned!
Tags: africa, cisco, malawi
Posted in Technology Projects, Trip | Comments Off
Wednesday, August 25th, 2010
I actually wrote this while I was in Malawi, forgot to post it until now though:
Earlier in my trip, while I was setting up Windows Sharepoint Services v3 for the folks at Tidziwe, I came across a problem that’s surprisingly common in the Sharepoint world. They wanted an employee directory on their intranet homepage. They also wanted department directories, with additional information in them on each of the department sites. In my mind, the perfect situation would be pulling this information from Active Directory. That way, the administrative effort is minimized, and there aren’t additional tasks that have to be done for each new user.
Unfortunately, the standard “People and Groups” list doesn’t work too well in this scenario because WSS only pulls the Active Directory attributes in once. If changes are made in the future to the Active Directory object, those changes won’t be reflected in Sharepoint. Also, instead of adding the AD user accounts directly in to the Sharepoint groups, I’m using existing AD groups as an intermediary. (Again, this should cut down on the extra administrative work that has to be done to give a user access – once they are added to the department group in AD, they should have access to Sharepoint, any file shares, etc.) The People and Groups list does not work at all in this case – it just shows the AD group that’s been added, and not the individual users. It’s kind of disappointing that Microsoft releases products that integrate so poorly with Active Directory.
So, realizing that an out-of-the-box solution was not going to work here I started to look at custom webparts, and other means with which I could achieve what I wanted. It appears that there are several expensive 3rd party webparts out there that try to fill this functionality gap. Still, I felt I could solve this quicker (and cheaper) on my own, so eventually I realized how I can do this.
Our department directories don’t necessarily have to be pulled from the Sharepoint groups, instead I felt that filtering the users based on the “Department” attribute in AD would be sufficient here. I realized that using DataViews in Sharepoint Designer can pull in data from a variety of sources, I eventually realized I could use a web service to pull the data from Active Directory and pass it in to the DataView.
I created a web service in ASP.NET that returns the user information (first/last name, department, email, phone, etc) from Active Directory. The web service has a couple methods, one returns all the Active Directory users, the other only returns the users from a particular department.
On the main employee directory, I have a DataView WebPart that uses the web service as its data source. It lists information for all users. I have it grouping the employees by their department to make it easy to find. Each department site also has a department directory for added convenience. These ones use the second method that just returns the employee information for a particular department. They both have clickable mailto: links for the email addresses.
 Sharepoint Employee Directory Altogether this seems to work pretty well, it does what it’s supposed to. There might be some performance issues in larger environments with more hits and more users since it doesn’t currently do any caching. That shouldn’t be too hard to add if needed however.
If any one is interested in this, send leave me a comment or send me an email. What we’re using can filter the Active Directory users based on the Department attribute, though I tested using groups to do the filtering as well.
Tags: africa, dotnet, malawi, sharepoint
Posted in Technology Projects, Trip | 3 Comments »
Tuesday, June 8th, 2010
Yesterday and today I’ve been installing and configuring Microsoft Exchange 2010 here. Right now this is just a pilot, for a few people here to try Exchange and see if its worth completing moving over to Exchange from AtMail, the email system that’s currently being used. To make life easier, I wanted to display all Active Directory users in the Global Address List (GAL), including users who don’t have Exchange mailboxes right now. Instead they have the Email attribute set on their user object in Active Directory.
Exchange will only include users who have been mail-enabled and users that have Exchange mailboxes in address lists (as well as groups, rooms, etc). In and of itself, this is frustrating because we now have to modify all the user accounts in AD to mail-enable them (even though they already have an email address attribute, there are additional Exchange attributes that have to be added to the object). To make it worse however, the Exchange Management Console GUI can only do this one user at a time, and it doesn’t use the existing email attribute. So (as far as I can tell) there is no built-in GUI way to mail enable large numbers of users.
Fortunately, since 2007 the Exchange management tools have been largely based around PowerShell. This makes it easy to script stuff like this. The get-user command will list all of the users from Active Directory. The enable-mailuser command will mail enable them. That part didn’t take long to figure out, though getting the commands to work took a little while. The email address has to be passed into the enable-mailuser command, but using the user’s email address attribute itself doesn’t work, it has to be converted to a string first.
The last part I figured out thanks to a post by Shay Levy on a Microsoft newsgroup that came up on Google: https://www.microsoft.com/communities/newsgroups/en-us/default.aspx?dg=microsoft.public.windows.powershell&tid=9b03c675-9b9c-431d-933b-4a00901c20c3&cat=en_US_3750E87B-4971-4A5C-A537-45F5D7ABBECC&lang=en&cr=US&sloc=&p=1. Someone there was asking how to do something similar to this with contacts instead of users.
Here’s the command I used below. At first, it would be best to start off testing with a single user:
get-user <username> | foreach { enable-mailuser $_ -externalEmailAddress $_.WindowsEmailAddress.toString() }
There are still some issues with this, it comes back with an error if the user has no email address specified in their AD object, and it also probably will come back with an error if the user is already mail enabled or if they have an Exchange mailbox, but it’s a start.
Tags: exchange, powershell, windows
Posted in Technology Projects, Trip | Comments Off
Friday, June 4th, 2010
One of the satellite clinics here in Lilongwe has enough users that it really should be connected to our main site with a site to site VPN. People are wanting to use Outlook for email, we need to be able to push out group policies, people should be able to log on to the Active Directory domain, etc. We had a client-server based VPN (OpenVPN), but it’s not easily possible to login to AD or apply group policies. Things would just be a lot simpler if there was connectivity between the LAN’s at both locations.
This was done in the past through another ISP here. It sort of worked, but they were extremely expensive. Unfortunately, instead of having quality hardware, we have several cheap DLink VPN routers. We probably would have had a lot fewer issues if they simply had Cisco routers or a Linux box at the remote site that we could use OpenVPN with, but unfortunately that’s not the case.
We were eventually able get them connected, though there seems to be no way to control what traffic it routes over the tunnel. With a Cisco site-to-site VPN, an access list could be used to control which traffic is tunneled and which traffic is not. The former service provider used a proxy to get around this and send all web traffic through our Internet connection, but man is that a ugly solution. Also, both the dynamic routing (with RIP) and SNMP monitoring “features” of these devices appear to do nothing at all.
Right now we’re having problems with traffic between the sites. ICMP works, but HTTP generally doesn’t. OpenVPN worked fine.
Right now, I think our problems are related to the MTU (maximum transmission unit), the maximum size of packet that a device can handle. Most 100 Mbps Ethernet equipment supports MTU’s of around 1500 bytes. This can frequently be a problem with VPN’s, because the VPN endpoints encapsulate the original packets inside of IPSec/GRE/OpenVPN/etc. packet. For an example, let’s assume that the MTU for path the VPN tunnel is traveling between its endpoints is 1500 bytes, and the overhead from the VPN is 50 bytes. If computer A sends a packet that’s 1480 bytes, and the VPN endpoint X encapsulates it, the resulting packet is over 1500 bytes. This means that the MTU for traffic going through the tunnel is less than the typical MTU because of the overhead caused by the VPN connection.
This can cause difficult to troubleshoot problems with a VPN tunnel, because some traffic, like normal pings/ICMP, that is well under the MTU will pass through the tunnel just fine, but larger packets can’t “fit” in the VPN tunnel.
Another interesting note however is that the DLink router works when its at another location, meaning that it could be related to the ISP’s configuration.
Tags: africa, cisco, dlink, vpn
Posted in Technology Projects, Trip | 2 Comments »
Tuesday, June 1st, 2010
Wow, had a really busy four or five days in Moshi, Tanzania. Wish I would have had more time to relax and sight see, it usually ended up being 12 hour days for us. We got a lot down, though we ran in to a lot of issues that slowed us down.
The major projects we worked on while we were in Tanzania were setting up a new VSAT Internet connection, installing a remote access system, installing a network monitoring system, managing the bandwidth utilization, as well as connecting group of data computers here to the rest of the network.
For the VSAT connection, I was responsible for modifying the Cisco router configuration to work with the new connection. It was mostly a matter of adding NAT entries, changing a couple IP addresses, and adding a new default route. We also helped a satellite technician with aligning the satellite dish and configuring the primary and backup satellite modems. That turned out to be the more challenging aspect. Almost all of our first day here was spent working on that. One of the biggest issues we ran in to was a mis-understanding between our service provider and the satellite installer/technician. The technician thought the angle he was given was the angle of elevation for the satellite dish, however it was apparently the angle of the beam as it hits the ground, which doesn’t take some factors in to consideration. The satellite modems we received were also tricky to install. Some of the settings we had to set could only be accessed in in particular mode, and switching between modes required a arduously slow reboot. Unfortunately, there is no way of simply copying a configuration file over to these modems, it all had to be done manually, going through a clunky menu-based null modem cable connection.
For the remote access, we ended up going with Windows Server PPTP VPN. I tried to configure the Cisco Easy VPN and Cisco WebVPN, though I wasn’t able to get them to work with the limited about of time we had here. So instead we looked at the Windows Server VPN. While it isn’t very customizable, it was much easier to get working than I thought it would be.
If we would have had a Linux machine available at that time, I would have chosen OpenVPN. That’s what I configured at the project in Malawi, that’s what I use at home. The day after doing the VPN, we did end up installing Linux to use on a server here to work as a transparent web proxy for Squid. On our Cisco gateway router, I used a route map to forward port 80 traffic through our Linux squid machine. This is the first time I’ve actually used Squid in this fashion, seems to work fairly well once I was able to work out a few kink in the configuration. Because of the limited bandwidth, we’re limiting the maximum download or upload size to 20 MB. There is a separate static IP set aside for cases where someone has to be allowed through the proxy. (It’s just another line on the Cisco ACL that chooses traffic to send through the proxy.) I wish there was a way to automatically disable the route map if the proxy machine was down though, that would make me more comfortable with this setup. I’ll try to post the configurations for some of this later when I have more time, I’m at an internet cafe in Nairobi now.
Update:
Here are the pertinent parts of the Squid configuration.
acl lan src 1.2.3.0/24
# Block some websites all the time
acl always_block url_regex -i "/etc/squid/always_block.txt"
http_access deny always_block
deny_info http://1.2.3.4:8080/blocked.html always_block
acl work_hours time MTWHF 08:00-17:00
# Block some websites only during work hours
acl workhours_block url_regex -i "/etc/squid/workhours_block.txt"
http_access deny workhours_block work_hours
deny_info http://1.2.3.4:8080/blocked.html workhours_block work_hours
# Block Malware
acl malware_block url_regex -i "/etc/squid/malware_block.txt"
http_access deny malware_block
deny_info http://1.2.3.4:8080/malware.html malware_block
# limit downloads to 20 MB, uploads to 20 MB
reply_body_max_size 20000000 allow lan
request_body_max_size 20 MB
http_access allow lan
http_access deny all
http_reply_access allow all
http_port 3129 transparent
And here’s the configuration on the router, that forces all the web traffic to go through the proxy (except for traffic from a few servers). Fa1 is the inside LAN interface.
int fa1
ip policy route-map WebFilterProxy
ip access-list extended WebFilterHosts
deny tcp any any neq www
deny tcp host 1.2.3.6 any
deny tcp host 1.2.3.5 any
deny tcp host 1.2.3.4 any
deny tcp host 1.2.3.3 any
permit tcp any any
route-map WebFilterProxy permit 5
match ip address WebFilterHosts
set ip next-hop 1.2.3.4
Tags: africa, cisco, vpn, vsat
Posted in Technology Projects, Trip | 2 Comments »
Wednesday, May 26th, 2010
While configuring Windows Sharepoint Services v3 here at the UNC Project in Malawi earlier this week, I encountered some really odd issues with groups. For ease of management, I have been basing the access to Sharepoint on Active Directory groups. Users were already members of groups, I just added those groups to Active Directory groups I created for Sharepoint. Those groups then correspond to groups in Sharepoint, that they are members of. However, the test user accounts I created that were members of the groups in Active Directory didn’t have access to Sharepoint, unless I manually added them to the groups in the Sharepoint site (defeating the purpose of using the Active Directory groups in the first place, this was supposed to make it easier to manage). In Active Directory, I also did not seem to be able to add any groups to a global group (or convert domain local groups to global groups / vice-versa), though this didn’t immediately appear to be related to me.
After several hours of troubleshooting, I eventually stumbled upon the help section for the different types of Active Directory Groups. There I learned that Active Directory does not supported nested global groups if it is in Windows 2000 Mixed Mode. I don’t think I have even seen a domain in Windows 2000 Mixed Mode, that was a “legacy” mode a decade ago. The Active Directory domain here (running Server 2003 on all the DC’s) was somehow still set to Windows 2000 Mixed Mode. Upping that to Windows Server 2003 Native Mode finally fixed my problem. According to the documentation, Windows 2000 Native Mode should work as well, but why continue to live in the past.
Tags: active directory, africa, malawi, sharepoint
Posted in Technology Projects, Trip | Comments Off
Sunday, May 23rd, 2010
On Thursday evening we finally were able to get connected to Tonse. We needed a public IP address so that people at the site in Area 18 an connect to our OpenVPN server. It was a long, tiring, journey to get that to work. At first, their modem / customer premise equipment (which is made by Greenpacket) was using NAT and acting as a router.
This complicated things, because our Cisco router also does NAT for our internal addresses, 10.10.13.10. We were originally attempting to use a DMZ setup, where the Tonse modem did NAT, but the outside IP of our Cisco router was in a DMZ so all traffic would go to it. This didn’t seem to actually work. To try to see where the problem was, we put a switch in between our Cisco router and the Tonse modem. When we opened the ports for OpenVPN on our Cisco router we were able to access it from the switch in between the modem and the router. However, even when putting the Cisco router in the DMZ for the Tonse modem, we weren’t able to access the VPN from outside of the Tonse modem.
Eventually we were able to enabled bridging mode on Tonse’s modem, so our Cisco router directly had a public 41.x.y.z IP address. This would make it much easier for the users at Area 18 to access our VPN server. NAT is only being done once.
There was still one caveat though. The router wasn’t routing the VPN traffic back out the interface to Tonse. I connected my laptop to the Tonse connection, and strangely enough, the default gateway my laptop received was not on the same subnet as the laptop itself. My laptop received an IP address on a 41.x.y.z subnet, however, the default gateway was a private 10.a.b.c IP. That still does not make much sense to me, but apparently it works. The router wasn’t able to get to ping that IP address. So I ended up adding a static route on the Cisco router, telling it to send traffic to the 10.a.b.c IP address through the Tonse interface. This allowed everything to work.
I also moved the OpenVPN service from our primary mail server to the backup mail server. At first it didn’t work, which was frustrating, as the configuration was identical. Then I realized that that I forgot to enable IP forwarding on the Linux server. After fixing that, it worked fine.
Michael went to Area 18 the next day, and they were able to connect to the VPN. It seemed to be a fair amount faster than it was when the connection was going through the Ariave satellite link. The latency from Michael’s house (also on Tonse, but on a different tower) seems to be greater than it is from Area 18.
Tags: africa, malawi, vpn, wimax
Posted in Technology Projects, Trip | Comments Off
|
