network – Mark Ma's Blog

Asymmetric routing and campus firewall

the problem

Despite Zhejiang University having a clear university-wide policy on domain registration, a Haining campus Information Technology Services (ITS) officer Z told me the campus has its own policy that refuses to follow due to 'security concern'. In other words, the Haining campus refuses to assist with University domain reverse proxy, such that any user who wants to have an Internet-accessible server will have to have its server located elsewhere. In my situation, it means an extra ￥2200/year cost to rent a VPS from ZJU main campus.

The actual scenario is, the Haining campus has internal connections with other ZJU campuses. In my situation, with the service still deployed on a local server, with a forwarded proxy from main campus VPS, any security breach would still happen locally, meaning the 'security concern' is nothing but a poorly made excuse.

Well, since the administrative board is not responsible for users in a typical Chinese-style organization, I will have to fix this issue by myself.

Interestingly, the reverse proxy can be actually configured from the main campus alone, and it should work without problem. So I circumvent the Haining campus ITS and registered a domain reverse proxy directly with the main campus.

The result is, strange. I noticed the server is indeed Internet accessible when applying for an HTTPS certificate, but local tests show that it can not be accessed from within China. In other words, it is accessible only from abroad.

the cause

I spent an entire afternoon trying to figure out why, and with hints from the main campus IT support and traceroute output, I finally discovered that the problem lies in asymmetric routing.

More detailly, all outgoing traffic from the Haining campus is redirected according to its destination. Traffic going abroad is going through the main campus exit, while others directly through the local campus exit. Now since the incoming traffic is all from the main campus entrance, this path is symmetrically routed, and the other one is not. The asymmetric routing will be blocked by the firewall due to DDOS attack prevention.

the fix

Now that I know the cause of the problem, the dillema arises that I still need Haining campus ITS's assistance to fix the issue. I need to make sure even those administrative roles like Z saw my request, they would not understand my true purpose. Considering Z's earlier response, she was either in an administrative position, or is really rookie in the network engineering field, both case I bet she would not want, or be designated to look into some issues too technical to understand.

A ticket is then sent to the Haining campus ITS to request all outgoing traffic of a specific ip to go through the main campus network gateway 'due to asymmetric routing' for 'research purpose, and to avoid incidental firewall block', together with some specified gateway ip I got from traceroute log. It turned out Z was not designated to the ticket.

Now that this is an existed routing, they accepted the reqest 'after requesting superior instruction'. Problem solved.

the bureaucracy

The is a typical bureaucracy 'there are policies and measures to counter' incident.

It turned out, I got lucky because incompetent officer like Z is indeed, incompetent. But this also means tragedy for all those who can not conceive a counter, and thus have to live with trash policies made by those incompetent officers.

On the other hand, trash policy actually works because not enough people know how to counter it, so the incompetent officer can remain in position.

To knowledge.

Campus network exploitation

original plan with exploit

The campus has strange regulations to restrict outbound speed but not LAN speed. The goal was to utilize every accessible computer with wired LAN access to speed up my campus outbound connections. By using load balancer over a large number of computers the outbound bandwidth is combined for multithread downloading. n2n is used to connect devices with admin access for easy management, direct v2ray is deployed to devices without admin access but with fixed ip and port access, and v2ray of reverse proxy is to deploy to devices without fixed ip and admin access.

new exploits

While I was examing the campus HPC I noticed something strange: the outbound speed is not limited (in terms of LAN speed) on the HPC. This is reasonable consider that many people are using the cluster. Furthermore, the HPC is shared with commercial users from outside the campus, and further inspection confirms that the HPC outbound is a direct link to local China Telecom network instead of China Education and Research Network. This is a major advantage as the former one is much more reliable and efficient.

First of all, I only have a user account on the HPC, meaning opening ports on firewall is out of the quesiton, let alone creating tun devices. A common practise under such situation would be creating a reverse proxy to another server.

However, a port scan shows local 90 and 8080 port is open but unused. Port 90 is unavailable to common users, but 8080 is totally accessible. This is unexpected as well as hilarious. Nginx and v2ray is then deployed with ws+tls+vmess to provide maximum security. Crontab is configured to ensure availablity across reboot.

sudo nmap -n -Pn -sS -p0-65535 [ip]

result

Due to the unlimited outbound speed of HPC the connection bottle neck is now the campus LAN. Ping to various servers drops significantly due to the advantage of China Telecom over CERNET. Secure connections are established. While all operations are under campus regulation, some part of the process should be seen as exploits.

potential counter measures

To counter the exploits campus net admin would have to:

properly configure firewall of public workstations
block all ports of LAN devices to avoid NAT traversal, which is totally unreasonable, or
restrict the LAN speed to outbound speed, which is equivelantly unreasonable

Before that the campus network outbound speed limit is merely a trash policy, an inconvenience, rather than a restriction. Considering the inconvenience inflicted, and the total legal and reasonable process, I hardly find any motivation to report such exploits.

Backup and synchronization summary

Disk failure costs. As a general rule, all valuable data should be kept in copy, preferably on different devices, in real time, and with 2-way sychronization support.

In the early days I had been using robocopy with batch script, rsync with shell script, even adb-sync with batch/shell scripts. Configurations were complex, the backup system was vulnerable to various compatibility problems (typically system locale problems and path naming restrictions). The backup was also not in real time, nor does these methods support 2-way sychronization due to the absense of a database.

Commercial solution costs, while a lot of open-source solutions do not work in a real-time fashion.

Syncthing is a great tool to solve the problems above. It even comes with bundled NAT traversal capability. A typical backup network shows as below,

where arrow shows backup direction, solid line represents user files, and dash line represents system files.

Some personal files are sychronized across devices as needed, while backup operations typically have the following ignore cases.

Linux user

.stignore at $HOME

/.cache
/.conda
/.config/syncthing/index*
/.local/share/tracker
/.nvm
/.npm
/.rbenv
/.vscode-server
/installation
/go
/repos/**/node_modules
/repos/**/node_cache
/**/cache

Linux root

.stignore at /

!/etc/nginx/nginx.conf
!/etc/nginx/conf.d
!/var/lib/postgresql
*

Windows user

.stignore at %userprofile%

\.vscode
!\AppData\Local\Packages\Microsoft.WindowsTerminal_8wekyb3d8bbwe\LocalState\settings.json
\AppData
\Documents\Tencent Files\*\TIM\Registry2.0.db
\Documents\Tencent Files\*\TIM\Registry2.0.db-journal
\Music
\notes
\OneDrive*
\repos\**\node_modules
\repos\**\node_cache
\Searches
\Videos
(?i)\NTUSER*

Proxy summary

I built a proxy network allowing me to access various resources across the Internet. As shown in the figure below, servers on different locations are deployed and connected by either direct proxy (solid line) or reverse proxy (dash line). connection mapping are labelled in the format of '(x to y) * z', indicating z groups of x clients each connecting to y servers.

by tools

v2fly/v2ray-core: secure proxy, balancer, router
- Qv2ray/Qv2ray: v2ray gui for pc
- SagerNet/SagerNet: v2ray gui for android
ntop/n2n: virtual LAN adaptor
alexkirsz/dispatch-proxy: adaptor combiner
nginx/nginx: server side http proxy, http reverse proxy, http port reuse
haad/proxychains: proxy chains, socks,http->app, for pc
proxifier: proxy chains, socks,http->app, for windows
ambrop72/badvpn/tun2socks: socks->adaptor
torproject/tor,tpo/applications/tor-browser: anonymous http
FelisCatus/SwitchyOmega: socks,http->browser

by application senario

bypass GFW with server port access

v2fly/v2ray-core+nginx/nginx: vmess+ws+tls->socks5, balancer, multiple domain cdn servers for acceleration and camouflage

bypass campus Internet outbound speed limit (campus has no LAN speed limit)

has LAN servers with server port access

v2fly/v2ray-core+nginx/nginx: vmess+ws->socks5, balancer, multiple in-campus servers for acceleration, speed *= number of servers

without servers

alexkirsz/dispatch-proxy: LAN + WLAN + USB LAN + USB hotspot: speed *= 4

reverse proxy without server port access

v2fly/v2ray-core: reverse proxy
microsoft/vscode: temporary port forwarding

LAN traversal with server root access

ntop/n2n

notes

campus LAN should be seen as public with even more strict censorship but less decryption ability, never use unencrypted proxy in campus LAN
watch out for DNS leak, use firewall rules to block port 53 if necessary
if necessary use firewall rules to block all connections except localhost proxy

Zhejiang University Network

Had a chat with one of the local campus network admin, here is some of what I learned about the Zhejiang University network

the entire lab building is merely connected with 2 1-Gbps links
the new campus WIFI is restricted to 30 Mbps as requested by old main campus leader 'to ensure consistency'
office table network switches are trash with only 100 Mbps link speed
private router can cause DHCP problems thus DHCP should always be turned off, or they will ban the mac address
private router is against regulation as security might be breached that some connection can not be traced
all records are preserved permanently for security purpose
all connections to campus network can be traced to someone who is responsible
- common link for sure can be traced to the account that is using it
- direct link (static ip) can be traced to the applicant
- most, if not all, direct link is bound to the device mac address
proxy service is detected using self developed software, through various techniques including using ip range pool