Blog

New Code Projects: Backblaze B2 Version Cleaner & VBA SharePoint List Library

Itā€™s been a while since Iā€™ve posted code of any description, but Iā€™ve been working on a couple of things recently that Iā€™m going to make publicly available on my GitLab page (and my mirror repository at code.jnf.me)

Backblaze B2 Version Cleaner

I wrote last week about transitioning my cloud backup to Backblazeā€™s B2 service, and I also mentioned a feature of it thatā€™s nice but also slightly problematic to me: it keeps an unlimited version history of all files.

Thatā€™s good, because it gives me the ability to go back in time should I ever need to, but over time the size of this version history will add up – and Iā€™m paying for that storage.

So, Iā€™ve written a script that will remove old versions once a newer version of the same file has reached a certain (configurable)Ā ā€œsafe age.ā€

For my purposes I use 30 days, so a month after Iā€™ve overwritten or deleted a file the old version is discarded. If I havenā€™t seen fit to roll back the clock before then my chance is gone.

Get the code here!

VBA SharePoint List Library

This one I created for work. Getting data from a SharePoint list into Excel is easy, but I needed to write Excel data to a list. I assumed thereā€™d be a VBA function that did this for me, but as it turns out I was mistaken – so I wrote one!

At the time of writing this is inĀ ā€œproof of conceptā€ stage. It works, but itā€™s too limited for primetime (it can only create new list items, not update existing ones, and each new item can only have a single field).

Out of necessity Iā€™ll be developing this one pretty quickly though, so check back regularly! Once itā€™s more complete Iā€™ll be opening it up to community contributions.

I have no plans to add functions that read from SharePoint to this library, but once I have the basic framework down that wouldnā€™t be too hard to add if youā€™re so inclined. Just make sure you contribute back!

Get the code here!

Blog

Windows Authentication on External Websites

My home network is domain-based, and Iā€™m running a Windows Server 2008 VM as the domain controller. Iā€™ve written
in the past
about how to use PHP to do authentication using domain
credentials, and that works great for some scenarios. As a case in point, I use
Pydio to host a web-based file manager that
allows me access to my files when Iā€™m out and about. Pydio runs on a linux
server VM on my home
server
, and it actually includes a built-in mechanism to authenticate
against an LDAP server (the Windows domain controller) so I didnā€™t have to
modify it with my PHP code. The principle is the same, though.

image

This is all good stuff for anything hosted on my home
server, but what if that isnā€™t what I want? What if I want to host something on
my external, public webserver, and still use my active directory credentials to sign in to
it? And, while weā€™re at it, what if I want to be even more restrictive and
limit access to a particular organizational unit within active directory?

As luck would have it, these are all problems that I solved
this week. Read on!

Creating a Reverse SSH Tunnel

The first thing we need is to establish a secure connection
between the external webserver (the VPS) and the internal webserver (the local
linux VM). Weā€™re going to use a reverse SSH tunnel to do this,
and, specifically, weā€™re going to use a tool called autossh that will keep an eye on
the tunnel and restart it if something goes wrong.

Iā€™ll skip most of the technical detail here, but essentially
a reverse tunnel is going to forward a particular port on the external server
to a particular port on the internal server. Itā€™s called a reverse tunnel
because itā€™s the internal server that triggers the connection. Thatā€™s important:
the internal server can reach the external one just fine, but not the other way
around (thanks to things like me having a dynamic IP address for my home
internet connection, my home routerā€™s firewall, DHCP, etc).

I installed autossh on my Ubuntu server VM:

sudo apt-get install autossh

And then wrote a one-line script that I placed in /etc/network/if-up.d:

#!/bin/sh
su -c "autossh -M 29001 -f -N -R 8080:localhost:80 remote-server.com" localadmin

Teasing this apart just a little, it uses the local account ā€œlocaladminā€
to run the command enclosed in the quotation marks. That command forwards port
8080 on ā€œremote-server.comā€ to port 80 on the local machine.

For this to work itā€™s essential that the user ā€œlocaladminā€
is able to log on to ā€œremove-server.comā€ without
needing to enter a password
.

The Local Server

The local webserver is where most of the heavy-lifting is
going to take place. The first thing I did was create a new virtual host in the
webserver configuration on that machine. Iā€™m using lighttpd, so my configuration looks like
this:

$HTTP["host"] =~ "^auth.gateway" {
    $HTTP["remoteip"] !~ "127.0.0.1" {
        url.access-deny = ("")
    }

    server.document-root = "/home/jason/WebServer/Production/auth.gateway"

    url.rewrite-once = (
        "^(.*)$" =>"auth.php"
    )
}

Essentially it creates a new host with the hostname ā€œauth.gatewayā€
thatā€™s only accessible to the local machine (127.0.0.1). Any request that comes
in regardless of the URI is rewritten to a file called auth.php in the document
root.

The hostname here isnā€™t real (i.e. thereā€™s no public DNS
entry for it), and thatā€™s probably a good thing for the sake of locking down
security as tightly as possible. Also on that line of thinking is the fact that
access is limited to the local machine. The webserver doesnā€™t know that requests
coming through our SSH tunnel are coming from another machine thanks to the
virtues of port forwarding ā€“ it thinks theyā€™re coming from other processes
running locally (i.e. coming from 127.0.0.1).

Next is auth.php itself, the engine that runs all this
stuff:

<?php
    session_name ('AGSESSID');
    session_set_cookie_params(600);
    session_start();

    $domain = "testdomain.local";
    $domaincontroller = "dc1.testdomain.local";
    $requiredou = "FamilyMembers";
    $authkey = "big-random-string";

    if (!isset($_SERVER['PHP_AUTH_USER']) || !isset($_SERVER['HTTP_X_AUTH_KEY']) || $_SERVER['HTTP_X_AUTH_KEY'] != 'big-random-string') {
        header('WWW-Authenticate: Basic realm="Authentication Required"');
        header('HTTP/1.0 401 Unauthorized');
        exit;
    } elseif (!isset($_SESSION['AuthKey']) || $_SESSION['AuthKey'] != md5($_SERVER['HTTP_X_FORWARDED_FOR'].$_SERVER['PHP_AUTH_USER'].$_SERVER['PHP_AUTH_PW'])) {
        $ldap = ldap_connect($domaincontroller);

        ldap_set_option($ldap, LDAP_OPT_PROTOCOL_VERSION, 3);
        ldap_set_option($ldap, LDAP_OPT_REFERRALS, 0);

        if (!$bind = ldap_bind($ldap, $_SERVER['PHP_AUTH_USER']."@".$domain, $_SERVER['PHP_AUTH_PW'])) {
            header('WWW-Authenticate: Basic realm="Authentication Required"');
            header('HTTP/1.0 401 Unauthorized');
            exit;
        } else {
            $result = ldap_search($ldap, "DC=".implode(",DC=", explode(".", $domain)), Ā "(samaccountname=".$_SERVER['PHP_AUTH_USER'].")");
            $entries = ldap_get_entries($ldap, $result);
 
            if (strpos($entries[0]['distinguishedname'][0], "OU=".$requiredou) === FALSE) {
                header('WWW-Authenticate: Basic realm="Authentication Required"');
                header('HTTP/1.0 401 Unauthorized');
                exit;
            } else {
                $_SESSION['AuthKey'] = md5($_SERVER['HTTP_X_FORWARDED_FOR'].$_SERVER['PHP_AUTH_USER'].$_SERVER['PHP_AUTH_PW']);
            }
        }
    }
?>

As you can see, this is quite a bit more sophisticated than
the previous example in my SSO post, but letā€™s tease this one apart at a high
level too.

When the script is loaded, it first checks to see if the thereā€™s
a username, password and ā€œAUTH_KEYā€ contained within the HTTP headers of the
request. If so, it verifies that the ā€œAUTH_KEYā€ is what it was expecting, and it
tries to use the username and password to establish an LDAP connection to the
Windows server VM. If thatā€™s successful, it retrieves some information about
the user and checks if theyā€™re a member of the required organizational unit.

If all that works it sends back an empty page, but,
crucially a HTTP 200 status (meaning everything is OK). If any of those checks fail
then it sends back a HTTP 401 status (unauthorized) header instead.

In addition to this, the script creates a PHP session with a
custom name. The name is custom to avoid collisions with any session that
the external server may be creating, but essentially the session lasts 10
minutes, and if subsequent requests come in for a user thatā€™s already been
authorized then it skips all the checks and just sends back the HTTP 200
header. It does that because without it, every HTTP request made to the remote
server (not just every page served, but every image on every page, every CSS
file, JavaScript file, etc, etc) would involve a query to the domain controller
to validate the credentials, and thatā€™s a potential bottleneck with performance
implications.

The Remote Server

My remote server is running nginx as its webserver (Iā€™m new to it,
but I think I like it better than lighttpd. Thatā€™s kind of beside the point
though). The configuration looks like this:

server {
    server_name example.com;
    listen 80;

    root /home/jason/example.com/www;

    auth_request /__auth;
    auth_request_set $saved_set_cookie $upstream_http_set_cookie;
    add_header Set-Cookie $saved_set_cookie;

    location = /__auth {
        auth_request off;

        proxy_pass http://localhost:8080;
        proxy_pass_request_body off;
        proxy_set_header Content-Length "";
        proxy_set_header Host auth.gateway;
        proxy_set_header X-Original-URI $request-uri;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Auth-Key "big-random-string";
    }
}

When we tease this one apart, there are two key details. One
is that ā€œauth_requestā€ declaration in the fourth line. As per nginxā€™s
documentation
, auth request ā€œimplements client authorization based on the
result of a subrequest.ā€ In other words, instead of nginx doing the
authentication itself it forwards the request on. The configuration above
defers the authentication of http://example.com to http://example.com/__auth.

The second crucial chunk of the configuration file is
everything within the ā€œlocation = /__authā€ declaration. The first thing this does
is turn off the ā€œauth_requestā€ functionality (otherwise, weā€™re stuck in an
endless loop), and it then creates a proxy to redirect any requests to
http://example.com/__auth to http://localhost:8080. Port 8080, if you recall,
is in turn forwarded through our SSH tunnel to port 80 on the local server.

Additionally, it sets the hostname involved in the request
to ā€œauth.gateway,ā€ forwards a couple of other pieces of information as headers,
and, for some added security, sends an ā€œX-Auth-Keyā€ header that our PHP script
checks for.

Back outside of this block the ā€œauth_request_setā€
declaration takes the session cookie from our PHP script and saves it to a
variable, then the following line (ā€œadd_headerā€) sends that cookie back to the
clientā€™s browser as part of the response they receive.

Done!

Real World Problems with All This

I mentioned earlier that we set a session cookie as part of
the whole interaction to avoid the need to query the LDAP server for every HTTP
request the remote server receives, and I said this was to avoid a performance
bottleneck. Thatā€™s true, but we still have a performance bottleneck here: for
every HTTP request the remote server receives itā€™s still querying the local
server through our SSH tunnel, even if all the local server is doing is
responding that the credentials are good based on the existence of the session
cookie.

This communication has to take place over my home internet
connection, which is absolutely not what my ISP intended it to be used for. I don’t think it’s against their terms and conditions or anything like that, itā€™s just that it isnā€™t
really fast enough.

If the site deployed on the remote server was one of my own
making then Iā€™d modify this approach to create an authentication API of sorts
on the local server, and Iā€™d do the session setting and credential caching
entirely on the remote server, drastically reducing the number of queries to
the local server (all the way down to a single query, in fact, when the session
is first created and the user logs on).

The other problem is one of securing the whole interaction.
Weā€™re using ā€œbasicā€ HTTP authentication methods here, which means that the
username and password are passed around in the clear (theyā€™re not encrypted or hashed as
part of the process). Thatā€™s necessary: the auth.php script has to receive the
password in cleartext because it has to pass it to the Windows server to check
its validity. Itā€™s also not an issue with the communication between the remote
and local webservers, because that happens through an SSH tunnel thatā€™s using
public/private keypairs to provide encryption. It is a problem for the
communication between the user and the remote webserver though, and it leaves
our user vulnerable in several ways (especially if theyā€™re using a public WiFI
hotspot). Essentially what Iā€™m getting it is that you must use SSL between the user and the remote server.

Conclusion

This is not the most optimal way of doing things,
particularly in regards to the bottleneck it creates by deferring authentication
of every HTTP request to my local server which is connected to the internet
using a typical consumer-grade ADSL connection, but as a quick and dirty way of
securing resources on my public webserver without needing to modify the
resource itself in any way, it works great!

Enjoy!

Blog

Compiling Third-Party Modules Into Nginx

I want my public web server, which runs nginx, to authenticate against my active directory server using LDAP. Iā€™ve written in the past about how to use PHP to authenticate against active directory in this way, but there are a couple of problems: my active directory server isnā€™t accessible to the internet, and I want to use standard HTTP authentication instead of username and password boxes included on a webpage.

The answer, I think, is to put an authentication PHP script on my home server, make that available to the public web server through an SSH tunnel, and then use nginxā€™s Auth Request module to authenticate against it using the public server as a proxy.

This is – I hope – less complicated than it sounds. Weā€™ll see, and Iā€™ll post more if and when Iā€™m successful, but the problem Iā€™ve initially run into is that nginx in Ubuntuā€™s repositories doesnā€™t include the Auth Request module. I have remove nginx and re-install it from source, compiling it with the additional module included.

Itā€™s a bit of a daunting process, but the page Iā€™ve linked seems like it will take me through it step by step.

Wish me luck!

Compiling Third-Party Modules Into Nginx

Blog

I’m Back!!

Did you miss me?

Hopefully you didnā€™t even notice I was gone, but two days ago Tumblr terminated my account, removing this blog and Shrapnel from the internet. I immediately contacted support as directed and heard back from them yesterday evening: my account had been closed for contravening Tumblrā€™s community guidelines in relation to spamming and affiliate marketing.

I replied to make the point that at no point have I engaged in spamming or affiliate marketing, and apparently someone there agreed because I am now back online. The issue, as it turns out, was that my two Tumblr blogs were sending visitors back to jason.jnf.me (where I had a script that presented the blog content in a subfolder, integrating it into the site to a much greater degree than a separate domain would).

In the short-term Iā€™ve removed the redirect by simply resetting my blogā€™s theme to the default, and Iā€™ll take some time on the weekend to restore the look and feel I had previously, and probably give each of them a custom subdomain.

In the longer term, I think itā€™s time to start looking for an alternative blogging platform. When it seemed as though all the content I had on this blog had disappeared I was extremely disappointed. I run my own server, so I probably shouldnā€™t be relying on third-party services anyway.

The obvious suggestion would be to install WordPress, and while that would work great for my blog content I think Iā€™d have a hard time implementing some of the other site pages on that platform. What I want is a CMS (to give me the ability to quickly and easily manage and edit content) that lets me build custom bits and pieces (like my feed page) on top of it. Iā€™ve chosen PyroCMS. Itā€™s built on the CodeIgniter framework that Iā€™ve previously used which should make for relatively easy extensibility. Itā€™s going to take me some time, but Iā€™ve installed it on my development server to start getting my hands dirty. Iā€™m just happy Iā€™m back online and I donā€™t have to spend this weekend trying to rebuild.

Blog

Single Sign-On (SSO) in PHP

Thereā€™s a project underway in work called single sign-on and identity and access management. Iā€™m not involved in it directly, although by its nature it touches on several things that I am working on at the moment. The goal, as the name implies, is to rid ourselves entirely of multiple sets of credentials: anything we use should have the same login ID and password, whether itā€™s one of our hosted systems (which, to be fair, already behave this way for the most part) or a third-party system like the webapp that we use to deliver training.

Since Iā€™m not directly working on it this project is not really anything more than a blip on my radar, but itā€™s interesting to me because Iā€™m attempting to do a similar thing at home, albeit on an entirely different scale to the large enterprise-wide project thatā€™s I hear about in my professional life.

After the recent upgrade to my home server that Iā€™ve blogged about before I now have several virtual servers included in our home network setup. One of these runs Windows Server 2008 R2, and Iā€™ve made that one a domain controller that all the other computers (and servers) connect to. There are several benefits to this approach, but chief amongst them is a single set of credentials ā€“ I use the same username and password regardless of which of our home computers Iā€™m logging on to, and when I change my password I change it once for it to be effective everywhere.

There are few web services running on our home network which require signing into, such as a web interface for centralized torrent downloads, a file browser, and a simple content management system that pulls everything together into an intranet of sorts. Most of these are PHP-based, and Iā€™m on a mission to add SSO capability to these too.

Iā€™ve discovered two main methods of enabling SSO in PHP that Iā€™ll write about after the break, and my eventual plan is to tie the two methods together into a single cohesive sign-on module that I can reuse. Read on to find out what Iā€™m up to!

LDAP (Lightweight Directory Access Protocol) Authentication

Wikipedia defines LDAP as an open, vendor-neutral, industry standard application protocol for accessing and maintaining distributed directory information services over an Internet Protocol (IP) network.

Thatā€™s a lot of fancy words for saying that LDAP provides an address book (think of the global address listing you see in Outlook, and youā€™re thinking of an LDAP database). PHP has a set of LDAP extensions that can be used to query the address book and retrieve user information, but in the context of authentication, we donā€™t even need to worry about any of that. An LDAP server can (depending on the implementation) be queried anonymously, or we can pass in some credentials with the query to get more detailed information back (again, depending on the implementation).

Itā€™s this last part thatā€™s important. Active Directory on a Windows domain controller is an LDAP server. In PHP, all we have to do is attempt to log on to the LDAP server. If weā€™re successful, itā€™s because the username and password that we input is valid on the domain. Even better, ā€œvalid on the domainā€ in this case means itā€™s an active account, the password is not locked, and all other account-level restrictions (such as a restricted set of logon hours) are considered.

All of this makes using LDAP to test the authenticity of a set of supplied credentials pretty trivial:

<?php
   $username = "testuser"
   $password = "pa55w0rd";

   $domain = "testdomain.local";
   $domaincontroller = "dc1.testdomain.local";

   $ldap = ldap_connect($domaincontroller);
   if ($bind = ldap_bind($ldap, $username."@".$domain, $password)) {
      // user login successful
   } else {
      // user login failed
   }
?>

Thatā€™s all there is to it!

Depending on what you had in mind when you read ā€œSSOā€ in the title of this post though, we may not have met your requirements here. If we meant that the user has a single set of credentials then, fantastic ā€“ they do! But if our intention was to only require that a user enters their single set of credentials once (when they log on to Windows) then weā€™ve fallen short here. The code above requires the username and plaintext password, so weā€™d have to present some kind of web-based login form to the user to request that information and get all this to work.

Enter NT LAN Manager (NTLM) Authentication

If a website (or intranet site) is part of the intranet or trusted zones (found in the Internet Settings control panel applet) then that site is allowed to pass a header requesting NTLM authentication. When it does, windows passes a header back containing some information about the currently logged-in user without the user being prompted for their credentials in any way.

I obtained some simple example code from someone called loune at Siphon9.net and modified so that it doesnā€™t require apache as the webserver. Hereā€™s the PHP:

<?php
   if (!isset($_SERVER['HTTP_AUTHORIZATION'])){
      header('HTTP/1.1 401 Unauthorized');
      header('WWW-Authenticate: NTLM');
      exit;
   }

   $auth = $_SERVER['HTTP_AUTHORIZATION'];

   if (substr($auth,0,5) == 'NTLM ') {
      $msg = base64_decode(substr($auth, 5));
      if (substr($msg, 0, 8) != "NTLMSSPx00")
         die('error header not recognised');

      if ($msg[8] == "x01") {
         $msg2 = "NTLMSSPx00x02x00x00x00".
            "x00x00x00x00". // target name len/alloc
            "x00x00x00x00". // target name offset
            "x01x02x81x00". // flags
            "x00x00x00x00x00x00x00x00". // challenge
            "x00x00x00x00x00x00x00x00". // context
            "x00x00x00x00x00x00x00x00"; // target info len/alloc/offset
            
         header('HTTP/1.1 401 Unauthorized')
         header('WWW-Authenticate: NTLM '.trim(base64_encode($msg2)));
         exit;
      }

      else if ($msg[8] == "x03") {

         function get_msg_str($msg, $start, $unicode = true) {
            $len = (ord($msg[$start+1]) * 256) + ord($msg[$start]);
            $off = (ord($msg[$start+5]) * 256) + ord($msg[$start+4]);
            if ($unicode
               return str_replace("", '', substr($msg, $off, $len));
            else
               return substr($msg, $off, $len);
         }

         $ntlm_user = get_msg_str($msg, 36);
         $ntlm_domain = get_msg_str($msg, 28);
         $ntlm_workstation = get_msg_str($msg, 44);
      }
   }

   echo "You are $ntlm_user from $ntlm_domain/$ntlm_workstation";
?>

Thereā€™s a big problem with this code, and the problem is that itā€™s just decoding the user information from the HTTP header, and assuming that all is good ā€“ thereā€™s no work done to confirm that the header is genuine, and there is a possibility that it could have been faked. We could do some tricks like confirming that the page request is coming from within our local network, but that doesnā€™t really solve the problem ā€“ HTTP headers can be manually defined by an attacker that knows what theyā€™re doing, and what weā€™re doing here is a bit like asking for a username and then just trusting that the user is who they say they are without doing any further authentication.

Combining the Two Approaches

Included in the NTLM authorization header that gets sent to the webserver during the passwordless authentication interaction described above is an MD4 hash of the userā€™s password. A newer version of louneā€™s code retrieves this and confirms its validity using samba. Unfortunately that setup wonā€™t work for me ā€“ my intranet webserver is running a customized version of samba that comes with the software I use to manage the linux computers that are attached to my domain, and this trick just flat-out fails.

However, if I have a plaintext version of the userā€™s password then I can use PHP to generate an MD4 hash of it for the purposes of comparison. So hereā€™s my plan:

Scenario A: The first time a user comes to my webapp weā€™ll get their credentials using NTLM, including the MD4 hash of their password. Since we wonā€™t know if this hash is valid, weā€™ll present the user with a screen asking them to confirm their password (but not their username). When they input it, weā€™ll confirm that their username and password combo is good using LDAP, and also generate an MD4 hash of the plaintext password that they entered to compare with what NTLM gave us. If nothing weird is going on everything should match. At this point weā€™ll store the MD4 password hash for future.

Scenario B: When a user returns to our webapp weā€™ll get their credentials using NTLM as before, and compare the hash NTLM gave us to our stored hash from their previous visit. If they match, weā€™re good, and thereā€™s no need to ask the user to enter their password.

Scenario C: If the NTLM hash and the stored hash donā€™t match then the most likely scenario is that the user has changed their Windows password since their previous visit to our webapp. In that case weā€™ll throw out the stored hash and start again at Scenario A.

If anyone knows of a better approach (is there a centrify Kerberos tool that I could use to get an MD4 hash of the userā€™s password for the purposes of my comparison, for example?) then please let me know! Iā€™d love to be able to achieve true passwordless SSO, but so far I canā€™t a method for doing so unless I switch my webserver from linux to Windows, and I donā€™t want to do that.