Linux File Server update!

The Linux file server itself continues to remain online and work well.

What proved less successful was regular interaction with it from a different operating system, macOS.

In principle, macOS and Linux impose few restrictions on file names. However, macOS Finder is less accommodating. Finder won't tolerate file names that contain the colon character. Worse, it won't even acknowledge that they exist.

This proved to be a day-to-day hassle. I was constantly having to "recover" files that I either had given a forbidden name from Linux, or whose names had caused problems when when I migrated them over from HFS Plus (macOS) to begin with.

So, I partially migrated to APFS (macOS). When the warranty expired on my 2012 Mac Mini, I installed a big extra hard drive in it, removed colon characters from all my important files, and copied them back to macOS. Then, I turned on SMB ("Windows") file sharing on my Mac Mini. Problem mostly solved.

I still keep media without DRM, which I consume primary thru Plex, on my Linux box. I also still use my Linux file system for most software development. If I need to keep code in sync with other operating systems, I use git. But I'm careful to use boring file names.

Software Development Misstep Artifacts

I've posted some artifacts from my recent software development missteps.

I initially planned to blog this here, but I know enough about Blogger not to even bother trying to cram big inline HTML5 videos into it.

Set up a Linux File Server

The Problem

Managing digital files isn’t easy these days. Improvements to media fidelity and the sheer volume of available content have arguably outpaced improvements in storage technology and affordability.

There are two important kinds of storage drives: hard disk drives (HDD) and solid state drives (SSD)

HDD’s have a magnetically sensitive disk spinning inside them. Specific data can only be read or written to when the portion of the disk it resides on is physically underneath a drive head. Since data is typically spread out, HDD's are slow.

Operating systems — especially commercial ones — have gotten big. Consequently, systems that run operating systems off of HDD’s are also slow. In fact, old Macs with just HDD’s can be almost unusably slow with the latest macOS installed.

SSD’s, on the other hand, have no mechanical parts. So you will probably want at least one of these in your system.

Unfortunately, they are significantly more expensive than HDD’s of the same capacity at the present time.

If you’re lucky, you can afford to buy an SSD that’s big enough to store all your files. But most of us can't. So unless you're sufficiently comfortable, patient and trusting to keep all of your files in the Cloud, that means putting together a system including both SSD's and HDD's (or, a hybrid or Fusion drive.)

Solution 1: String Together Lots of External HDD's

The first draft of the solution might be to add HDD's to a single computer with an internal SDD.

Unfortunately, if you're a Mac user with recent hardware and you're too squeamish to void your warranty by cracking (and in most cases, I do mean cracking) them open, you'll need external drive enclosures for those HDD's.

But there are problems with these things.

1. They are noisy.
2. They must be attached with a forest of cables (See my blog post on Meta-Towers for a partial workaround)
3. Unless you leave your system on all the time, they have to be turned on and off separately from the computer they're paired with.

Mac SSD's have been enjoying the new Apple File System (AFS) since the release of macOS High Sierra. But Mac HDD's are currently still stuck on the old HFS+ file system. HFS+ lacks checksumming to guard against file corruption.

If you're aren't Mac user, file management is easier. But you'll still encounter frustrations if you want to use your files remotely, from other devices (including from other computers.)

Solution 2: File Servers

One solution is to set up a file server.

A file server is a computer that lets other devices read and edit its files remotely, using high-level networking protocols.

There are several options for file server software. I find that the best ones are significantly easier to set up on Linux. I'll discuss the options in the next section.

Linux and the EXT4 file system are great at reliably and flexibly serving files. They are also free!

(Note that, unlike macOS, no mainstream Linux file system offers a Versions feature that consistently lets you revert to earlier versions of a file.)

Your file server could also be your primary computer system. But most people don't want Linux for their primary systems, and dual boot can be difficult to set up and keep set up.

So I recommend getting a cheap, expandable Linux tower with lots of internal drive bays. I am very happy with my own zaReason Limbo. I actually have mine rigged up to share a display with my Mac mini, and to share a clicky Unicomp keyboard via a USB 2.0 switch.

It's up to you how many internal HDD's to get and how big they should be. One 3 TB HDD holds pretty much all my permanent files except for archived iMovie projects, star catalogues, and planetary object video. (If you want to get technical, I suppose I also have Nintendo games on other drives.)

You'll need to pick a Linux distribution and a desktop environment. Ubuntu is my distribution of choice, and I use KDE as my main desktop environment (I got them packaged together as "kubuntu".) But I also like Gnome.

You'll want to set up mounts for all your installed HDD's so they're available immediately upon startup. You can do this by editing /etc/fstab, or by writing startup scripts that invoke udisksctl mount.

File Server Software

Save yourself a headache and install file server software using a package manager (kubuntu includes Muon). And be sure to keep up to date with the latest versions, to protect against security exploits.

After installation, you'll need to locate the configuration file for your new file server software. Typically, you can accomplish this with Linux's locate shell command, or find / -name ….conf. If there's more than one search result, you might need to experiment to determine which is the one your service is using.

Package managers usually configure sensible and secure defaults. But just be aware that a misconfigured file server in principal could expose all your files to the local network — or even the Internet — without requiring a password or anything! So take the time to review your configurations and understand what they mean and do.

Your package manager should set up your file server software as a Linux service (usually, a systemd service) that you can stop, start and check the status of with the service shell command.

The file server softwares discussed below aren't mutually exclusive. If you want, you could install all or them.

SMB via Samba

Samba is a free and open source implementation of Server Message Block (SMB) protocol. SMB has traditionally been associated with Microsoft Windows, but today is pretty much everywhere.

As of SMB version 3.0, Apple now recommends its use for file sharing rather than AFP (see below). But many users complain about SMB 3.0'd performance.

Samba Configuration

Samba's configuration file is typically called smb.conf and its service is typically called smb.

Once you've found the configuration file, edit it.

Under the [global] section, make sure passdb backend = tdbsam

(It's fine if there's also a colon and a file path after this.)

TDB is an encrypted authentication store that performs well enough for our purposes.

Remove the ; comment character in from of the [homes] section. Do the same for browsable, and change the value to yes. Also remove the comment character from read only, and change it to no, if necessary.

Save the file and restart your Samba (again, probably "smb") service.

Unfortunately, Samba is not well integrated into Linux's own authentication mechanism. You'll need to create Samba user accounts. You will typically accomplish this with the smbpasswd -a command.

You can avoid the need to reconfigure if you use the same names for your Linux and Samba accounts. 

You may find it helpful to troubleshoot your Samba configuration by installing and running smbclient.

smbclient -U your_user_name //your_host_name/your_user_name -d 3

where -d 3 indicates the debug level (the higher the level, the more verbose.)

AFP via Netatalk

Netatalk is a free and open source implementation of Apple Filing Protocol (AFP) which has been supported on Macs since version 9 of the classic Mac OS. As a result, Netatalk can probably talk to any Mac you still have in operation.

Configuration

Netatalk typically has a configuration file named "afp.conf" and its service is typically called "netatalk".

Configuring Netatalk is easier than Samba.

Create or uncomment a homes section in the configuration file.

[Homes]
      basedir regex = /home

You can also create sections for any other important directories.

[Another]
      path = /my_user_account_name/media/my_volume
      valid users = my_user_account_name

By default, recent versions of Netatalk allow the use DHX2 and DHCAST128 encryption by default. That's fine, just as long as cleartext isn't enabled. You can check on this using the asip-status.pl script.

Unlike with Samba, you'll be able to login with your existing Linux user account.

tail -f Netatalk's log file (its location should be configured) and then restart the Netatalk ("afp") service. If you don't see anything worrying in the logs, then you should be good to go.

Other Options

There are still other file server options.

NFS

Network File System (NFS) protocol originated on Solaris, but today is viewed as the native file server protocol of Linux.

NFS isn't quite as widely supported outside the macOS and Unix family of computers, but it has been battle-tested and is pretty fast.

SFTP via Open SSH Server

Secure File Transfer Protocol (SFTP) sits on top of the ssh protocol and service. 

Ssh and SFTP let you exploit public key infrastructure to securely serve files without ever needing to type a password.

On Mac, the excellent Transmit app ships with Finder plugins to seamlessly mount SFTP file systems as volumes.

WebDAV

Web Distributed Authoring and Versioning (WebDAV) piggybacks file sharing atop a webserver. It can be installed into Apache webserver (httpd).

Connecting to Your File Server

Please note that you will not need to enable anything in the Sharing control panel on your Mac… these options are solely for incoming connection requests.

To initiate an outgoing connection to your file server, pull down the Go file menu from the Finder and select "Connect to Server…"

Here's how to construct your connection string for each protocol:

Protocol	Format
Samba	smb://server_name/share_name
AFP	afp://server_name/optional_share_name
SFTP	sftp://server_name/optional_directory_path
NFS	nfs://server_name/optional_directory_path

Once in, things may appear a little weirder than you're used to. For example, old custom Finder icon files appear as "I7CIPB~N" over SMB. But for the most part, everything should be OK. 

Some folks in the online community recommend improving SMB 3.0 performance by reconfiguring your Mac to change the SIGNING_ON client setting to FALSE. But this could expose you to eavesdropping attacks if someone infiltrates your network, so I recommend against it. If you're concerned about performance, don't initiate SMB 3.0 connections from macOS Sierra or High Sierra.

Also keep in mind that if you need to do batch operations on a lot of large files, you always have the option of doing so on your Linux server, without using a file sharing protocol. For this, you could shell in with ssh or mirror its screen with Remote Desktop Client (RDC). (Because of Linux's relative lack of reliance on modifier key / mouse combinations, RDC from Mac to Linux works a lot better than the other way around.)

Or, of course, you could use your Linux box directly, like a normal system.

Search

Oh wait, what about search? I don't think about it often, but when I do, I consider it to be pretty much the most useful thing on my computer.

Linux is well supported in this area by Recoll and Gnome Tracker. They install easily with a package manager. Both can handle important common file types like PDF, ePub, and HTML.

Recent versions of Gnome Tracker can even be integrated with Netatalk and Samba! Unfortunately, there don't seem to be package manager packages for these versions at the present time, so you'd have to compile them manually, from source code.

But if Linux isn't your primary operating system, then chances are you use some native apps with idiosyncratic file formats. On Mac, these apps often ship with automated plugins to that allow Mac's search tool, Spotlight, to index their files. (I understand there is something similar called Wox on Windows.)

Linux won't be able to exploit these custom formats.

The solution is to let Spotlight index your remote volumes, just like it indexes the files residing on your Mac's internal drive! Sure, the initial index will be slow to build over your network. But that index will live on your primary system's SDD, so the actual searches will be fast.

Unfortunately, indexing of remote drives from macOS has been slightly broken since the release of macOS Sierra. But a simple fix to create /private/var/db/Spotlight-V100/Volumes/ (if missing) will allow you to enable a remote volume:

mdutil -i on /Volume/volume_name

Spotlight indexes are protocol and volume specific. So you won't be able to connect via, say, AFP and exploit an index made over SMB.

I kind of like maintaining Spotlight and Gnome Tracker indexes. That way, I can validate them against each other.

Backup

You'll want to the schedule regular back ups of your file server. I find the easiest way to do this is with an HDD dock. They make it easy to rotate backup drives.

On Linux, Back in Time is a good backup solution.

For what it's worth, I've found that initial backups complete much faster on EXT4 than on HFS+.

Whatever you pick, don't forget to keep using it and every now and then test with a spare drive to make sure you're really backing up what you believe you are.

Serving Other Things

If you're new to Linux, you'll find it's good at serving lots of other things, too.

You might want to install a dedicated media server like Plex. There's a client app for Plex on Apple TV, Roku, and Amazon Fire.

If you want to serve webpages to your local network, try Apache Webserver (httpd), which you may already have if you went the WebDAV route for file serving. Nginx is another good option.

You may even want to run a database server. There are many, many options. PostgreSQL and MySQL are good go-to picks.

Kid Coaster

I built a kid-sized roller coaster in the basement! Here's the story.

Pulling Out of the Station

For the 2013 holidays, I had constructed a kid-sized railroad train for my daughter. I adapted the front car from a Radio Flyer Classic Walker Wagon and the rear car from an older model by replacing the wheels with CW4 U-wheels from CamOnWheels. The track consisted of two long PVC pipes, tied periodically by metal clamps affixed to wood beams.

This train was pretty cool. Thanks to ball bearings in the wheels, it moved smoothly and was satisfying to operate. But it could only go in a straight line, and it couldn’t go very fast without risking derailment.

Next Stop: Kid Coaster

I started thinking about the holiday train’s successor. I had been impressed by homemade roller coasters I'd seen online. Could I build a small one?

I started researching roller coaster construction. I studied the photographs at coastergallery.com and johnnyupsidedown.com. I read books and coasterbuzz.com forum posts.

There were still a lot of gaps to fill in, but I began to think I might be able to pull it off.

Design

I decided to build the roller coaster in the basement, so I could work at night and keep the resulting clutter to a minimum.

Wheels and Wheel Assemblies

Roller coasters have several wheels. Some of those wheels are positioned at right angles to one another, to keep the roller coaster on its tracks as it follows a curved trajectory.

These wheels are typically grouped into assemblies.

Wheel assemblies on pro wooden roller coaster cars are usually metal. I didn't have convenient access to welding equipment, so I chose wood reinforced with steel bolts. I'd attach four of these assemblies to my vehicle.

For the actual wheels, I selected nylon inline skate wheels. Four of these wheels are strong enough to carry an adult. I'd also use four, but design my loaded vehicle to weigh a lot less than an adult.

The wheels would need to accommodate at least the track thickness, which I set at 19 mm, based on available boards.

I fired up SimpleCAD and came up with the simplest design I could. I iterated on it a couple of times until arriving at this.

The guide wheels on the bottom and side never carry the full load of the vehicle. As a result, they can safely hug the track, exposing it to the brunt of their modest force. The top wheels, on the other hand, typically do carry the full weight (or close to the full weight) of the loaded vehicle. Consequently, they would sit on a laminated track support.

The wheels are all bolted-in, and the bolts also serve as their axles. The side wheel's axle has to be in a different plane than the top and bottom wheels' axles. But since there are four wheel assemblies, the forces all balance out.

On the diagram above, light gray is steel and dark gray is nylon. The olive rectangles are bolts and the red rectangles are inline skate spacers, which brace the wheel bearings. Most everything else is wood.

The dash-outlined rectangles in the wood on the left side indicate pocket holes. They permit the wheel assemblies to sit flush against the vehicle, without interference from the bolt heads.

The gray-brown areas on the righthand drawing indicate drill bores.

Bolts would be secured with Nylock nuts at their ends (not depicted.)

Vehicle

The vehicle itself would basically just be a crate long enough for a kid to sit in. I'd assemble it from a series of wood slats mutually braced and held together by screws.

Tracks

Although some newer pro wooden coasters do have prefab metal tracks, most are still wood.

Some homemade coasters use tracks made out of PVC pipes that have been heated, and then bent. But that produces toxic gas. I was also worried that cyclic strain would lead to mechanical failure over time.

I made the decision pretty early on to go with wood. Wood is strong, forgiving and easy to work with.

As much as I intially wanted to attempt a banked turn, the available space wouldn't really allow it. I eventually settled on a track path that would trace out a valley between two peaks in a single vertical plane.

The vehicle would hang from its four wheel assemblies, threaded around the track. I picked a track support gauge (the distance between the two tracks) of 69 cm, to accommodate a typical crate plus a margin.

I managed to find an industry handout on safety inspections that contained isometric drawings of the “ledgers” that tracks and track supports are built atop. I basically copied that design. It would need multiple reinforcing beams to safely support a 23 kilogram passenger at a factor of four margin of safety.

Wherever possible I spread out loads in this manner, to eliminate single failure points.

The design for the pitched track trajectory would need to be informed by subsequent testing. I ended up revising it many times over the duration of the project.

Sourcing and Testing

I found most of the hardware I needed at the hardware store. An exception was the 10 cm long steel bolts for the top wheels, which also proved expensive. Luckily I only needed four of them.

The lumber for the track and understructure came from my local Home Depot. Its quality proved uneven enough that I resolved to use a wholesaler on my next project. I learned to sift through entire lumber piles and revisit after restockings to find usable timbers.

Originally, I had planned to use published product specs to determine how much load each component could handle. But I rarely was able to find any. Instead, I ended up having to conduct my own tests.

For example, to ensure that the vehicle’s slatted wood bottom was up to 4 times the required load, I built a prototype and proceeded to jump up and down on it. It held.

The track surface and support laminates would need to be bent to assume the shape of the valley. I hoped to accomplish this without the need to resort to steaming or notching.

With essentially no prior woodworking experience, I was worried about bending the track timbers into place with clamps. How far could I safely bend them? If I bent them too far, would they suddenly snap, assaulting me with splinters and chunks of broken wood?

So my original plan was to use thin laminate strips, which I figured would be easy to bend.

They were. In fact, experiments showed I could safely bend them significantly further than I had supposed. Since the required number of strips would have been expensive, I decided instead to go with thicker “2 by” timbers. These did necessitate the purchase of special, monster-sized bending clamps, but cooperated afterward.

Luckily, further, cautious experiments suggested that I couldn’t supply enough force by hand to break “2 by” timbers (but don’t try this at home!)

So the track support ultimately consisted of a stacked laminate of pine nominal 2 by 4 by 10’s. I would secure them with a portable electric drill and staggered deck screws.

Finally, I'd cover the ledger surface with bent peg boards.

Construction

I started in on the wheel assemblies. It was hard to get completely straight cuts with my Japanese hand saw. Even after buying an engineer’s square, I ended up rebuilding the wheel assemblies several times to get them right.

My design also involved very long bores that I kept screwing up with a hand drill. So I got a drill press. That helped, but the drill press didn't always have enough vertical clearance when its support tray was installed. So occasionally I had to replace the tray with a makeshift sub-base and/or drill from both ends, trying my best to meet in the middle.

Lengthwise, the track surface consists of two separate boards and I could never manage to get completely smooth, first-order continuity at their contact point. But, to my surprise, testing revealed it not to be a serious problem.

I needed many deck screws to complete the track understructure. I learned the hard way to consistently drill-out adequate holes for them ahead of time.

I'd always been too lazy to build a sawhorse for previous projects. During construction on this project, the incomplete ledger often served as one. Slightly inclined, but extremely secure.

Once the peg boards were installed, I found I could actually lie down on the ledger!

Maintaining wheel clearance along the entire length of the bent track required some contour adjustments and a lot of shaving. But once everything was planed down and the vehicle was reinstalled, it felt correct.

Safety

I loaded the vehicle up with weights and did a test run. To my relief, it worked well.

I couldn’t figure out how to attach seat belts to the vehicle without compromising its structural integrity. So instead, I opted for a padded restraint bar. I screwed wooden socket pairs into both sides of the otherwise symmetric vehicle, to accommodate different sized-kids. The dowel bar could be secured in these ports with a clamp.

Inspired by the styling on deluxe Radio Flyer wagons, I made some cushions from textured Denier Nylon Oxford fabric, quilt batting, and Muppet foam. For the outside I assembled a sort of fabric tube, which was tricky to feed through the sewing machine. Generous pinning helped.

Then I hot glue gunned the foam and batting together. I inverted the fabric tube and slid it over. I folded the ends up like wrapping paper and sewed them up by hand.

With a full maximum load, the vehicle nearly reached the other opposite peak after release. Nevertheless, it was important that the vehicle not run off the end of the track, even if inadvertently overloaded or pushed.

So I built cradles into the ledger on each end that could accept vertical stop beams. This way, I'd be able to easily remove the vehicle for repairs when needed. I duct-taped a bunch of Muppet foam to the beams, to dissipate the force of any impact.

Launch

Kid Coaster has been in regular operation for a month now!

I tend to think the best way to understand how something works is to build one for yourself. I do feel like I understand roller coasters better now, but really I just have more specific questions.

My next woodworking projects will be to build a derby racer (with my daughter) and a boat. But I find I'm having a hard time bringing myself to dismantle the roller coaster track, to make room.

Meta-Towers

After learning that the dead graphics card in my 2011 iMac would cost $700 to replace (!), I needed a new Mac desktop system.

The iMac had been a solid computer overall, but its fancy monitor was now effectively useless to me (older iMacs do have a "Target monitor mode" that allows them to be used as an external display for another computer, but only if the old iMac is still bootable.)

I didn't want to end up in that situation again, so iMacs were out.

I considered a "trash can" Mac Pro, but didn't like its cost or power utilization.

Much has recently been written about the Mac Mini's three years without an upgrade. I reluctantly decided to buy one anyway, with 3GHz and an SSD. I'm glad I did! The SSD really makes it feel zippier than any desktop Mac I've previously owned.

(I recently recounted my experience getting Adobe CS3 up and running on it.)

But the computer configuration I had really wanted all along was a tower, like older Mac Pros, and like the zareason Limbo 5880 Linux file server I've migrated most of my files to.

So, I decided to assemble a Meta-Tower for my new Mac Mini, using a Mesh Stainless Steel Stackable Legal Size Letter Paper Tray Desktop Organizer. You may want to get one from The Container Store, as the ones on Amazon seem to have a tendency to arrive a little bent.

The Meta-Tower co-locates the Mini with its peripherals and preserves some degree of physical access, without seriously obstructing waste heat dissipation.

The same general idea, of course, could be applied to an Intel NUC, a Raspberry Pi, or any other compact, self-contained computer.

The Mac Mini is snuggled on the bottom tier, alongside a router.

The middle tier has an external hard drive ("HDD"), for Time Machine backups.

The top tier has a USB 3.0 hub (which has proved to be largely unnecessary as recent Minis have a generous collection of USB ports.) It also hosts a couple of HDD docks, which I use for archiving astrophotography video. They let me swap HDD's in and out, almost as easily as Flash drives.

I configured the devices so things I'd need to access (buttons & ports) faced forwards, or at least were near the open front. The Meta-Tower sides were partially obstructed, but not so much that it prevented me from threading cords through.

I expected I'd need to physically secure everything in place, but it turned out the cords were rigid enough to do that relatively well. Well-paired devices also braced one another.

Where possible, I plugged power cables into a power strip on the floor beneath the desk. Devices that need to be plugged in and unplugged alot have a separate power strip atop my desk.

I tried to use the shortest cords I could find. Lengths of 3 feet (91 cm) are definitely better than 6 feet (183 cm)!

I built another Meta-Tower for my desktop game consoles.

The bottom tier holds my Nintendo Wii. My Retro Freak needed to be on top, so I could continue plugging game cartridges into it from above and access its front-facing game control ports. The top tier also houses a classic controller adapter, which has additional ports.

I've used a Brother labeler to distinguish some cartridge slots and cables, and an HDMI switcher to tie it all together.

I'm not a laptop fan, but I have to use one for work. I reserved the third port on my HDMI switcher for it. The laptop itself won't fit in a desktop organizer, so instead it lives atop a Rain Design mStand Laptop Stand.

Meta-Towers have some limitations. Unlike a real tower, the satellite devices don't all switch on and off in concert with their associated computer.

As mentioned, it can be difficult or impossible to access devices that aren't on the topmost of a meta-tower.

And of course there still are quite a few visible cords.

But when all is said and done, Meta-Towers have reduced the amount of clutter on my desks. I also find them conceptually helpful, as organizational units.

TARDIS Roundel Combination Bulletin/Peg- board

I'm happy to report that my TARDIS Roundel Combination Bulletin/Peg- board is still going strong!