Upgrading to Windows 10, Part 2

I took the plunge on upgrading on my main machine.  These were the hiccups:

  1. I have a DisplayLink USB/DVI adapter for an external monitor.   The “compatibility check” Windows runs prior to upgrading failed to catch that I needed to install the latest driver.  It just was a blank screen after upgrading.  So I had to get it manually from here and fortunately it seems to be work (otherwise I’d have to rush out and buy a new adapter).
  2. Vagrant somehow forgot the network interface it had been using prior and didn’t find the network interfaces installed.  Upon up, it asks “default: which interface should the network bridge to?” which is described in this issue.  So I had to upgrade VirtualBox to the latest 5.0.x test build (which is semi-compatible with Windows 10), and upgrade Vagrant to v1.7.4 to work with it.  Then upon reboot and relaunch of Vagrant I was able to pick the interface I wanted (although it still seems to forget my selection each time).
  3. My PuTTY keys quietly disappeared from the Pageant key list and had to be re-added manually.  Seem to work now though.
  4. TortoiseGit overlay icons all disappeared as per this issue which appears to point to OneDrive (which I have no intention of using) as the culprit.  I followed the steps in my comment there to work around it.


Still open: Windows Search was not indexing any program directories, so it couldn’t find any applications at all!  So I manually added all the Program directories, and we’ll see if it works.


Upgrading to Windows 10

I updated my laptop from Windows 8.1 to Windows 10 over the last day.  I did not want to leave this major update up to Microsoft’s timing, so I installed all the available updates from Windows Update, which triggered the little Windows 10 icon at the bottom of the screen.  I then initiated the upgrade via the steps at this post.

Here’s what I learned:

  1. Be patient, enormously so.  I found that that several of the dialog windows can realistically be “spinning” for several hours or more – even after you have supposedly downloaded the several GB’s of update files.
  2. Startup times seem much slower than Windows 8.1.  Don’t know if this will improve over time, or not.
  3. Upon boot, I saw several DOS windows flash across the screen with error messages relating to OneDrive, which I have never used.  I also don’t believe I have configured a Windows Live account for this laptop in the past either (I only use a local account).  I do see an taskbar icon for OneDrive so clearly Microsoft expects me to be using that already!
  4. I got a notification that my “Wifi Sense” needed attention.  But after clicking it, all I got was a blank Settings screen.  I have no interest in the (rather creepy) Wifi Sense feature.  Fortunately it will only apply to unsecured networks so I don’t think it represents a security risk on my home networks.
The good news is all my applications and files seem to be untouched.  I’ll give it a few more days and if it seems stable I will take the plunge on my primary machine.

Disinfecting a hacked WordPress site

Recently I took over management of a WordPress site for a nonprofit that had been hacked at some point in the previous few months, and my job was to recover it.  It had been hosted at the time on GoDaddy and it appeared that there had been a GoDaddy-specific exploit that the hackers had known of.

Pretty quickly I found the hackers’ PHP files (for malware, it appeared) in both the webroot and under /wp-content/uploads – so those were quickly deleted.  I thought that was it, but I was wrong.

The site was running a somewhat-dated version of WordPress, maybe 3.9 – so it was 6-12 months old, which isn’t too bad.  It used a hodgepodge of plugins though, and I was afraid to upgrade it for fear of breaking some plugin (I had not budgeted any debugging time to this).  So I didn’t upgrade WordPress core, but should have in retrospect: as it turned out, there was also a spam-relay PHP hack under /wp-includes that I had missed.  When I got the site up on a new locked-down server, I didn’t notice at first that the postfix queue was getting absolutely hammered with outbound mails from this domain – which wasn’t hosting email in any case.

So I was in a situation where there were dozens of mails entering the queue per second.  I had no idea where they were coming from, as this server wasn’t even the destination of the MX records for the domain.  So I did the following:

  1. Remove the MX records, and all associated A records (like email, smtp, webmail, etc.) from the zone entirely.  Now in hindsight I doubt it would have had any impact in this case, but if you’re not running mail on a domain, you don’t need any of that.
  2. Check the mail queue via mailq and individual mails via postcat.  There’s more detail on syntax as this link.  You can delete the entire queue via

    nice postsuper -vv -d ALL
  3. To be safe, I also set up postfix to reject all emails being sent from this domain, by following the steps here.

Eventually I thought to look at the access logs and found the hackers’ requests to /wp-includes which obviously is wrong.  I then did a clean reinstall of the WordPress core via git, and the outbound spam attempts ceased.

Moral of the story, always do a clean reinstall of the core.

Migrating email to Google Apps, 2015 edition

Did another email migration, about 50K emails, to Google Apps last week, this time from a very ancient mail system on Verio.  A few more notes to add from this time around:

  1. First step was to lower the TTL to 600 on the MX records.
  2. Verio offers both an IMAP and POP interface, with ports documented here.
  3. The Verio username to authenticate the email account is the person’s mailbox name (e.g., “joe”).  It is not the username used to log into the Verio control panel.  This wasn’t documented anywhere.  If you use the latter, you can actually set up an IMAP account in Outlook with it, send and receive mails successfully (from joe’s account no less!), but no mail is synced.  So once I figured this out, I could set up an IMAP account and downloaded all 50K emails, 3GB worth, over the course of a day.
  4. However none of the official migration tools from Google worked for various reasons:
    1. The Data Migration Service was the first option, but failed since the Verio server lacks a valid SSL certification.  Verio also uses usernames (not email addresses) to authenticate IMAP, which the tool doesn’t allow.
    2. Then tried GAMME, but never could configure it to authenticate with this ancient mail server.  Looks like it was using Dovecot but kept returning garbage.
    3. Next option was Mail Fetcher – but that doesn’t work since in this case we’re fetching from the same address, which isn’t allowed.  If the MX records still point to the old server I don’t see why they’d prevent this since that seems like a perfect scenario for this feature.
    4. Then went to Google Apps Migration for Microsoft Outlook – however, this only works on PSTs (generated by a POP account) and not the OSTs used by IMAP accounts.  This too, wasn’t documented.  So I had to spend another day syncing to Outlook via a new POP account.  So I then launched the tool but unfortunately, I could only get ~ 100 mails to sync at a time before it errored out.  So I gave up on this.
  5. This left me with only the “Outlook copy-paste” option.  So I set up an IMAP account in Outlook for Google Apps, created a new folder in it, unsubscribed all other folders I could, disabled scheduled Send/Receive, copied all 50K emails from the Verio POP account to this, and then let it Synchronize.
  6. Unfortunately Outlook throws an ugly error dialog and halts the synchronization when it tries to sync a message that’s bigger than 25MB so I had to keep manually restarting it.  Hopefully this is fixed in the next version of Outlook so it can continue in the background.
  7. But a few days of this and everything finally made it up to the Google Apps folder.  I then logged into the webmail interface, and moved all the mails to the Inbox and deleted the temporary label.
  8. Last, I switched the MX records to point to Google’s and raised the TTL to a more normal 3600.

Looks like it was all successful, and the email is in a much better, more reliable, and more secure system now.

Updating an SSL certificate on Plesk Parallels

I seem to keep forgetting the steps here, when I have to re-do it every few years.

  1. Create a CA bundle file, in the proper order, via the steps here.
  2. In the Plesk dashboard (tested in v9.5.4), go to Home->Domains->yourdomain.com->SSL Certificates, and add a new one.  Paste in, or upload your certificate file for “Certificate” and the bundle file for “CA Certificate”.  I think the first time through, you may have to add in the private key file as well.
  3. Go up a level to Web Hosting Settings and select the new certificate you just made.
  4. Go to Home->Services Management, and stop Apache and then start it.  This step may be extraneous.
  5. You’re not done yet!  Now put the bundle file, certificate, and private key somewhere safe on the server filesystem.  The first two may now be in /usr/local/psa/var/certificates/ but it’s not clear they are ever used – at any rate we’ll ignore them.
  6. Update /etc/httpd/conf.d/ssl.conf with the following:
    # This seems to override whatever is in the domain VirtualHost.
     SSLCertificateFile /path/to/domain_com.crt
     SSLCertificateKeyFile /path/to/domain.com.key
     SSLCACertificateFile /path/to/domain_com.ca-bundle
  7. Now bounce Apache again.  You’ll see the new certificate.

If you run into trouble you can check this out for details on how to verify the SSL certificate being sent.