Set up a Linux File Server

The Problem

Managing digital files isn’t easy these days. Improvements to media fidelity and the sheer volume of available content have arguably outpaced improvements in storage technology and affordability.

There are two important kinds of storage drives: hard disk drives (HDD) and solid state drives (SSD)

HDD’s have a magnetically sensitive disk spinning inside them. Specific data can only be read or written to when the portion of the disk it resides on is physically underneath a drive head. Since data is typically spread out, HDD's are slow.

Operating systems — especially commercial ones — have gotten big. Consequently, systems that run operating systems off of HDD’s are also slow. In fact, old Macs with just HDD’s can be almost unusably slow with the latest macOS installed.

SSD’s, on the other hand, have no mechanical parts. So you will probably want at least one of these in your system.

Unfortunately, they are significantly more expensive than HDD’s of the same capacity at the present time.

If you’re lucky, you can afford to buy an SSD that’s big enough to store all your files. But most of us can't. So unless you're sufficiently comfortable, patient and trusting to keep all of your files in the Cloud, that means putting together a system including both SSD's and HDD's (or, a hybrid or Fusion drive.)

Solution 1: String Together Lots of External HDD's

The first draft of the solution might be to add HDD's to a single computer with an internal SDD.

Unfortunately, if you're a Mac user with recent hardware and you're too squeamish to void your warranty by cracking (and in most cases, I do mean cracking) them open, you'll need external drive enclosures for those HDD's.

But there are problems with these things.

1. They are noisy.
2. They must be attached with a forest of cables (See my blog post on Meta-Towers for a partial workaround)
3. Unless you leave your system on all the time, they have to be turned on and off separately from the computer they're paired with.

Mac SSD's have been enjoying the new Apple File System (AFS) since the release of macOS High Sierra. But Mac HDD's are currently still stuck on the old HFS+ file system. HFS+ lacks checksumming to guard against file corruption.

If you're aren't Mac user, file management is easier. But you'll still encounter frustrations if you want to use your files remotely, from other devices (including from other computers.)

Solution 2: File Servers

One solution is to set up a file server.

A file server is a computer that lets other devices read and edit its files remotely, using high-level networking protocols.

There are several options for file server software. I find that the best ones are significantly easier to set up on Linux. I'll discuss the options in the next section.

Linux and the EXT4 file system are great at reliably and flexibly serving files. They are also free!

(Note that, unlike macOS, no mainstream Linux file system offers a Versions feature that consistently lets you revert to earlier versions of a file.)

Your file server could also be your primary computer system. But most people don't want Linux for their primary systems, and dual boot can be difficult to set up and keep set up.

So I recommend getting a cheap, expandable Linux tower with lots of internal drive bays. I am very happy with my own zaReason Limbo. I actually have mine rigged up to share a display with my Mac mini, and to share a clicky Unicomp keyboard via a USB 2.0 switch.

It's up to you how many internal HDD's to get and how big they should be. One 3 TB HDD holds pretty much all my permanent files except for archived iMovie projects, star catalogues, and planetary object video. (If you want to get technical, I suppose I also have Nintendo games on other drives.)

You'll need to pick a Linux distribution and a desktop environment. Ubuntu is my distribution of choice, and I use KDE as my main desktop environment (I got them packaged together as "kubuntu".) But I also like Gnome.

You'll want to set up mounts for all your installed HDD's so they're available immediately upon startup. You can do this by editing /etc/fstab, or by writing startup scripts that invoke udisksctl mount.

File Server Software

Save yourself a headache and install file server software using a package manager (kubuntu includes Muon). And be sure to keep up to date with the latest versions, to protect against security exploits.

After installation, you'll need to locate the configuration file for your new file server software. Typically, you can accomplish this with Linux's locate shell command, or find / -name ….conf. If there's more than one search result, you might need to experiment to determine which is the one your service is using.

Package managers usually configure sensible and secure defaults. But just be aware that a misconfigured file server in principal could expose all your files to the local network — or even the Internet — without requiring a password or anything! So take the time to review your configurations and understand what they mean and do.

Your package manager should set up your file server software as a Linux service (usually, a systemd service) that you can stop, start and check the status of with the service shell command.

The file server softwares discussed below aren't mutually exclusive. If you want, you could install all or them.

SMB via Samba

Samba is a free and open source implementation of Server Message Block (SMB) protocol. SMB has traditionally been associated with Microsoft Windows, but today is pretty much everywhere.

As of SMB version 3.0, Apple now recommends its use for file sharing rather than AFP (see below). But many users complain about SMB 3.0'd performance.

Samba Configuration

Samba's configuration file is typically called smb.conf and its service is typically called smb.

Once you've found the configuration file, edit it.

Under the [global] section, make sure passdb backend = tdbsam

(It's fine if there's also a colon and a file path after this.)

TDB is an encrypted authentication store that performs well enough for our purposes.

Remove the ; comment character in from of the [homes] section. Do the same for browsable, and change the value to yes. Also remove the comment character from read only, and change it to no, if necessary.

Save the file and restart your Samba (again, probably "smb") service.

Unfortunately, Samba is not well integrated into Linux's own authentication mechanism. You'll need to create Samba user accounts. You will typically accomplish this with the smbpasswd -a command.

You can avoid the need to reconfigure if you use the same names for your Linux and Samba accounts. 

You may find it helpful to troubleshoot your Samba configuration by installing and running smbclient.

smbclient -U your_user_name //your_host_name/your_user_name -d 3

where -d 3 indicates the debug level (the higher the level, the more verbose.)

AFP via Netatalk

Netatalk is a free and open source implementation of Apple Filing Protocol (AFP) which has been supported on Macs since version 9 of the classic Mac OS. As a result, Netatalk can probably talk to any Mac you still have in operation.

Configuration

Netatalk typically has a configuration file named "afp.conf" and its service is typically called "netatalk".

Configuring Netatalk is easier than Samba.

Create or uncomment a homes section in the configuration file.

[Homes]
      basedir regex = /home

You can also create sections for any other important directories.

[Another]
      path = /my_user_account_name/media/my_volume
      valid users = my_user_account_name

By default, recent versions of Netatalk allow the use DHX2 and DHCAST128 encryption by default. That's fine, just as long as cleartext isn't enabled. You can check on this using the asip-status.pl script.

Unlike with Samba, you'll be able to login with your existing Linux user account.

tail -f Netatalk's log file (its location should be configured) and then restart the Netatalk ("afp") service. If you don't see anything worrying in the logs, then you should be good to go.

Other Options

There are still other file server options.

NFS

Network File System (NFS) protocol originated on Solaris, but today is viewed as the native file server protocol of Linux.

NFS isn't quite as widely supported outside the macOS and Unix family of computers, but it has been battle-tested and is pretty fast.

SFTP via Open SSH Server

Secure File Transfer Protocol (SFTP) sits on top of the ssh protocol and service. 

Ssh and SFTP let you exploit public key infrastructure to securely serve files without ever needing to type a password.

On Mac, the excellent Transmit app ships with Finder plugins to seamlessly mount SFTP file systems as volumes.

WebDAV

Web Distributed Authoring and Versioning (WebDAV) piggybacks file sharing atop a webserver. It can be installed into Apache webserver (httpd).

Connecting to Your File Server

Please note that you will not need to enable anything in the Sharing control panel on your Mac… these options are solely for incoming connection requests.

To initiate an outgoing connection to your file server, pull down the Go file menu from the Finder and select "Connect to Server…"

Here's how to construct your connection string for each protocol:

Protocol	Format
Samba	smb://server_name/share_name
AFP	afp://server_name/optional_share_name
SFTP	sftp://server_name/optional_directory_path
NFS	nfs://server_name/optional_directory_path

Once in, things may appear a little weirder than you're used to. For example, old custom Finder icon files appear as "I7CIPB~N" over SMB. But for the most part, everything should be OK. 

Some folks in the online community recommend improving SMB 3.0 performance by reconfiguring your Mac to change the SIGNING_ON client setting to FALSE. But this could expose you to eavesdropping attacks if someone infiltrates your network, so I recommend against it. If you're concerned about performance, don't initiate SMB 3.0 connections from macOS Sierra or High Sierra.

Also keep in mind that if you need to do batch operations on a lot of large files, you always have the option of doing so on your Linux server, without using a file sharing protocol. For this, you could shell in with ssh or mirror its screen with Remote Desktop Client (RDC). (Because of Linux's relative lack of reliance on modifier key / mouse combinations, RDC from Mac to Linux works a lot better than the other way around.)

Or, of course, you could use your Linux box directly, like a normal system.

Search

Oh wait, what about search? I don't think about it often, but when I do, I consider it to be pretty much the most useful thing on my computer.

Linux is well supported in this area by Recoll and Gnome Tracker. They install easily with a package manager. Both can handle important common file types like PDF, ePub, and HTML.

Recent versions of Gnome Tracker can even be integrated with Netatalk and Samba! Unfortunately, there don't seem to be package manager packages for these versions at the present time, so you'd have to compile them manually, from source code.

But if Linux isn't your primary operating system, then chances are you use some native apps with idiosyncratic file formats. On Mac, these apps often ship with automated plugins to that allow Mac's search tool, Spotlight, to index their files. (I understand there is something similar called Wox on Windows.)

Linux won't be able to exploit these custom formats.

The solution is to let Spotlight index your remote volumes, just like it indexes the files residing on your Mac's internal drive! Sure, the initial index will be slow to build over your network. But that index will live on your primary system's SDD, so the actual searches will be fast.

Unfortunately, indexing of remote drives from macOS has been slightly broken since the release of macOS Sierra. But a simple fix to create /private/var/db/Spotlight-V100/Volumes/ (if missing) will allow you to enable a remote volume:

mdutil -i on /Volume/volume_name

Spotlight indexes are protocol and volume specific. So you won't be able to connect via, say, AFP and exploit an index made over SMB.

I kind of like maintaining Spotlight and Gnome Tracker indexes. That way, I can validate them against each other.

Backup

You'll want to the schedule regular back ups of your file server. I find the easiest way to do this is with an HDD dock. They make it easy to rotate backup drives.

On Linux, Back in Time is a good backup solution.

For what it's worth, I've found that initial backups complete much faster on EXT4 than on HFS+.

Whatever you pick, don't forget to keep using it and every now and then test with a spare drive to make sure you're really backing up what you believe you are.

Serving Other Things

If you're new to Linux, you'll find it's good at serving lots of other things, too.

You might want to install a dedicated media server like Plex. There's a client app for Plex on Apple TV, Roku, and Amazon Fire.

If you want to serve webpages to your local network, try Apache Webserver (httpd), which you may already have if you went the WebDAV route for file serving. Nginx is another good option.

You may even want to run a database server. There are many, many options. PostgreSQL and MySQL are good go-to picks.

Kid Coaster

I built a kid-sized roller coaster in the basement! Here's the story.

Pulling Out of the Station

For the 2013 holidays, I had constructed a kid-sized railroad train for my daughter. I adapted the front car from a Radio Flyer Classic Walker Wagon and the rear car from an older model by replacing the wheels with CW4 U-wheels from CamOnWheels. The track consisted of two long PVC pipes, tied periodically by metal clamps affixed to wood beams.

This train was pretty cool. Thanks to ball bearings in the wheels, it moved smoothly and was satisfying to operate. But it could only go in a straight line, and it couldn’t go very fast without risking derailment.

Next Stop: Kid Coaster

I started thinking about the holiday train’s successor. I had been impressed by homemade roller coasters I'd seen online. Could I build a small one?

I started researching roller coaster construction. I studied the photographs at coastergallery.com and johnnyupsidedown.com. I read books and coasterbuzz.com forum posts.

There were still a lot of gaps to fill in, but I began to think I might be able to pull it off.

Design

I decided to build the roller coaster in the basement, so I could work at night and keep the resulting clutter to a minimum.

Wheels and Wheel Assemblies

Roller coasters have several wheels. Some of those wheels are positioned at right angles to one another, to keep the roller coaster on its tracks as it follows a curved trajectory.

These wheels are typically grouped into assemblies.

Wheel assemblies on pro wooden roller coaster cars are usually metal. I didn't have convenient access to welding equipment, so I chose wood reinforced with steel bolts. I'd attach four of these assemblies to my vehicle.

For the actual wheels, I selected nylon inline skate wheels. Four of these wheels are strong enough to carry an adult. I'd also use four, but design my loaded vehicle to weigh a lot less than an adult.

The wheels would need to accommodate at least the track thickness, which I set at 19 mm, based on available boards.

I fired up SimpleCAD and came up with the simplest design I could. I iterated on it a couple of times until arriving at this.

The guide wheels on the bottom and side never carry the full load of the vehicle. As a result, they can safely hug the track, exposing it to the brunt of their modest force. The top wheels, on the other hand, typically do carry the full weight (or close to the full weight) of the loaded vehicle. Consequently, they would sit on a laminated track support.

The wheels are all bolted-in, and the bolts also serve as their axles. The side wheel's axle has to be in a different plane than the top and bottom wheels' axles. But since there are four wheel assemblies, the forces all balance out.

On the diagram above, light gray is steel and dark gray is nylon. The olive rectangles are bolts and the red rectangles are inline skate spacers, which brace the wheel bearings. Most everything else is wood.

The dash-outlined rectangles in the wood on the left side indicate pocket holes. They permit the wheel assemblies to sit flush against the vehicle, without interference from the bolt heads.

The gray-brown areas on the righthand drawing indicate drill bores.

Bolts would be secured with Nylock nuts at their ends (not depicted.)

Vehicle

The vehicle itself would basically just be a crate long enough for a kid to sit in. I'd assemble it from a series of wood slats mutually braced and held together by screws.

Tracks

Although some newer pro wooden coasters do have prefab metal tracks, most are still wood.

Some homemade coasters use tracks made out of PVC pipes that have been heated, and then bent. But that produces toxic gas. I was also worried that cyclic strain would lead to mechanical failure over time.

I made the decision pretty early on to go with wood. Wood is strong, forgiving and easy to work with.

As much as I intially wanted to attempt a banked turn, the available space wouldn't really allow it. I eventually settled on a track path that would trace out a valley between two peaks in a single vertical plane.

The vehicle would hang from its four wheel assemblies, threaded around the track. I picked a track support gauge (the distance between the two tracks) of 69 cm, to accommodate a typical crate plus a margin.

I managed to find an industry handout on safety inspections that contained isometric drawings of the “ledgers” that tracks and track supports are built atop. I basically copied that design. It would need multiple reinforcing beams to safely support a 23 kilogram passenger at a factor of four margin of safety.

Wherever possible I spread out loads in this manner, to eliminate single failure points.

The design for the pitched track trajectory would need to be informed by subsequent testing. I ended up revising it many times over the duration of the project.

Sourcing and Testing

I found most of the hardware I needed at the hardware store. An exception was the 10 cm long steel bolts for the top wheels, which also proved expensive. Luckily I only needed four of them.

The lumber for the track and understructure came from my local Home Depot. Its quality proved uneven enough that I resolved to use a wholesaler on my next project. I learned to sift through entire lumber piles and revisit after restockings to find usable timbers.

Originally, I had planned to use published product specs to determine how much load each component could handle. But I rarely was able to find any. Instead, I ended up having to conduct my own tests.

For example, to ensure that the vehicle’s slatted wood bottom was up to 4 times the required load, I built a prototype and proceeded to jump up and down on it. It held.

The track surface and support laminates would need to be bent to assume the shape of the valley. I hoped to accomplish this without the need to resort to steaming or notching.

With essentially no prior woodworking experience, I was worried about bending the track timbers into place with clamps. How far could I safely bend them? If I bent them too far, would they suddenly snap, assaulting me with splinters and chunks of broken wood?

So my original plan was to use thin laminate strips, which I figured would be easy to bend.

They were. In fact, experiments showed I could safely bend them significantly further than I had supposed. Since the required number of strips would have been expensive, I decided instead to go with thicker “2 by” timbers. These did necessitate the purchase of special, monster-sized bending clamps, but cooperated afterward.

Luckily, further, cautious experiments suggested that I couldn’t supply enough force by hand to break “2 by” timbers (but don’t try this at home!)

So the track support ultimately consisted of a stacked laminate of pine nominal 2 by 4 by 10’s. I would secure them with a portable electric drill and staggered deck screws.

Finally, I'd cover the ledger surface with bent peg boards.

Construction

I started in on the wheel assemblies. It was hard to get completely straight cuts with my Japanese hand saw. Even after buying an engineer’s square, I ended up rebuilding the wheel assemblies several times to get them right.

My design also involved very long bores that I kept screwing up with a hand drill. So I got a drill press. That helped, but the drill press didn't always have enough vertical clearance when its support tray was installed. So occasionally I had to replace the tray with a makeshift sub-base and/or drill from both ends, trying my best to meet in the middle.

Lengthwise, the track surface consists of two separate boards and I could never manage to get completely smooth, first-order continuity at their contact point. But, to my surprise, testing revealed it not to be a serious problem.

I needed many deck screws to complete the track understructure. I learned the hard way to consistently drill-out adequate holes for them ahead of time.

I'd always been too lazy to build a sawhorse for previous projects. During construction on this project, the incomplete ledger often served as one. Slightly inclined, but extremely secure.

Once the peg boards were installed, I found I could actually lie down on the ledger!

Maintaining wheel clearance along the entire length of the bent track required some contour adjustments and a lot of shaving. But once everything was planed down and the vehicle was reinstalled, it felt correct.

Safety

I loaded the vehicle up with weights and did a test run. To my relief, it worked well.

I couldn’t figure out how to attach seat belts to the vehicle without compromising its structural integrity. So instead, I opted for a padded restraint bar. I screwed wooden socket pairs into both sides of the otherwise symmetric vehicle, to accommodate different sized-kids. The dowel bar could be secured in these ports with a clamp.

Inspired by the styling on deluxe Radio Flyer wagons, I made some cushions from textured Denier Nylon Oxford fabric, quilt batting, and Muppet foam. For the outside I assembled a sort of fabric tube, which was tricky to feed through the sewing machine. Generous pinning helped.

Then I hot glue gunned the foam and batting together. I inverted the fabric tube and slid it over. I folded the ends up like wrapping paper and sewed them up by hand.

With a full maximum load, the vehicle nearly reached the other opposite peak after release. Nevertheless, it was important that the vehicle not run off the end of the track, even if inadvertently overloaded or pushed.

So I built cradles into the ledger on each end that could accept vertical stop beams. This way, I'd be able to easily remove the vehicle for repairs when needed. I duct-taped a bunch of Muppet foam to the beams, to dissipate the force of any impact.

Launch

Kid Coaster has been in regular operation for a month now!

I tend to think the best way to understand how something works is to build one for yourself. I do feel like I understand roller coasters better now, but really I just have more specific questions.

My next woodworking projects will be to build a derby racer (with my daughter) and a boat. But I find I'm having a hard time bringing myself to dismantle the roller coaster track, to make room.